Assessment of CMIP6 model performance for temperature and precipitation in Xinjiang, China

Xiolu Zhng , Lijun Hu , Dng Jing , ,

a College of Earth and Planetary Sciences, University of Chinese Academy of Sciences, Beijing, China

b Institute of Atmospheric Physics, Chinese Academy of Sciences, Beijing, China

Keywords:CMIP6 Evaluation Temperature Precipitation Xinjiang

ABSTRACT In this study, the authors evaluate the skill of 42 climate models from phase 6 of the Coupled Model Intercomparison Project (CMIP6) in reproducing the climatological temperature and precipitation in Xinjiang during the period 1995—2014. The results indicate that the models can reliably capture the geographical distributions of the two variables. The regionally averaged bias of temperature is 0.1°C for the annual mean, ? 1.6°C in spring,0.5°C in summer, ? 0.2°C in autumn, and 1.3°C in winter. Regionally averaged annual and seasonal precipitation in Xinjiang is generally overestimated by the CMIP6 models. The simulated annual precipitation is 89% more than the observation over Xinjiang, with a regionally averaged bias of 256% in spring, ? 3% in summer, 84% in autumn, and 258% in winter. Quantitative analysis indicates that most models overestimate the spatial variability of both climatological temperature and precipitation. The models show smaller discrepancies in simulating the temperature than the precipitation in Xinjiang. In comparison, both the median and arithmetic mean of the 42 models have similar skills to those of 29 selected good models, and outperform most individual models.

1. Introduction

Xinjiang is located in the hinterland of Eurasia, accounting for approximately one sixth of China’s total land area. It is a typical arid and semi-arid region over Asia, with limited precipitation and intense evaporation. The climate in Xinjiang is highly sensitive to climate change owing to its unique geographic location and mountain—basin system ( Wang et al., 2020a ; Wu et al., 2010 ). Previous studies indicate that, because global warming has accelerated the water cycle, increased precipitation, and intensified evaporation, the climate in Xinjiang has experienced significant “warming—wetting ” changes since the mid-1980s ( Shi et al., 2002 , 2007 ). Both the annual surface air temperature and precipitation increased significantly over there, though certain differences exist among regions and seasons ( Ren and Yang, 2006 ;Chen et al., 2009 ; Li et al., 2011, 2016 ; Wang et al., 2017 ; Wu et al.,2019a ). From 1961 to 2018, the increasing trend of the surface air temperature was 0.3—0.4°C/10 yr over Xinjiang ( Wang et al., 2020a ),which is higher than over China and over the globe ( Li et al., 2012 ).The changes in precipitation have widely impacted the agriculture,fragile ecological environment, and people’s livelihoods in Xinjiang( Zhang et al., 2003 ; Zhao et al., 2010 ). As stated in the Fifth Assessment Report of the Intergovernmental Panel on Climate Change(IPCC), global mean surface air temperature could have increased by 1.5°C or even 2°C by the end of the 21st century ( IPCC, 2013 ).As such, climate change over Xinjiang has attracted increasing attention.

Global climate models have been widely applied to simulate the climate in the past, present, and future. They are built on the fundamental laws of physics, fluid dynamics, chemistry, and so on. Current models are able to reproduce the large-scale patterns of historical surface air temperature and precipitation ( Flato et al., 2013 ). However, models are still imperfect because, for instance, of our limited understanding of the real climate system, the problems that arise from computing nonlinear formulae to represent nature, and adopting inadequate parameterizations for physical processes. Thus, it is necessary to evaluate the ability of climate models from various perspectives before using them to carry out scientific studies.

Previous studies have evaluated the models that participated in phase 5 of the Coupled Model Intercomparison Project (CMIP5) in simulating the surface air temperature and precipitation over China as a whole (e.g., Xu and Xu, 2012 ; Chen and Frauenfeld, 2014 ; Chen et al.,2014 ; Wu et al., 2015 ; Jiang et al., 2016 ; Zhang et al., 2016 ) or just the Qinghai—Tibetan Plateau (e.g., Su et al., 2013 ; Hu et al., 2014 ; You et al.,2016 ). It was found that models could reasonably capture the geographical distribution of these features. However, relatively less attention was paid to Xinjiang in these earlier evaluation studies. According to previous assessments of CMIP5 models at global and national scales, warm biases and excessive precipitation exist in Xinjiang ( Xu and Xu, 2012 ;Chen and Frauenfeld, 2014 ; Zhao et al., 2014 ). The models involved in the latest phase, phase 6, of CMIP (CMIP6) have improved relative to CMIP5 in terms of their simulation of surface air temperature and precipitation for the whole of China ( Wu et al., 2019b ; Guo et al., 2020 ;Jiang et al., 2020a ; Wyser et al., 2020 ; Zhou, 2020 ; Zhou et al., 2020 ;Zhu et al., 2020 ; Zhang et al., 2021b ). However, the skill of CMIP6 models in reproducing the climate in Xinjiang remains unclear.

In this study, we evaluate the ability of 42 CMIP6 models in reproducing the annual and seasonal surface air temperature at 2 m (hereinafter simply referred to as temperature) and precipitation in Xinjiang.In section 2 , we introduce the data and methods. Section 3 describes the observed and simulated climatological temperature and precipitation in Xinjiang and their differences. Conclusions are presented in section 4.

2. Data and methods

2.1. Data

The performances of the 42 CMIP6 models in simulating the climate in Xinjiang are evaluated using the historical simulations forced by observed time-evolving external forcings ( Table 1 ; Eyring et al.,2016 ). Monthly temperature and precipitation from the first ensemble run of each model are analyzed, covering the period 1995—2014(same as the definition in Jiang et al. (2020b) , Lovino et al. (2021) ,and Zhang et al. (2021a) ). Further details are available online at https://esgf-node.llnl.gov/projects/cmip6/ .

Table 1 Description of the 42 CMIP6 climate models employed in this study.

The observed monthly temperature and precipitation from CN05.1,established by the Chinese National Climate Center, are used to evaluate the models. The variables are achieved by interpolating data from 2416 meteorological observation stations in China, and the data have a horizontal resolution of 0.5° × 0.5° ( Wu and Gao, 2013 ).

2.2. Methods

All the model data were remapped to the observed horizontal resolution of 0.5° × 0.5° by using bilinear interpolation. The seasonal means of spring, summer, autumn, and winter are defined as the average of March—April—May, June—July—August, September—October—November,and December—January—February, respectively.

Considering that the interpolated temperature errors are usually in proportion to local elevation and topographical complexity, a “topographic correction ” process needs to be carried out, especially in western China ( Zhao et al., 2008 ). The correction of temperature was performed using the following equation:

whereT′ andTdenote the corrected and the interpolated temperature (°C), respectively;Zmodrepresents the surface altitude (m) of each model, whileZCN05.1refers to that of CN05.1;γis the lapse rate of ? 6.5°C km?1; andirepresents each grid point (700 points in Xinjiang in total).

From a quantitative perspective, the spatial correlation coefficient(SCC), the ratio of standard deviation (RSD), and the normalized centered root-mean-square error (CRMSE) between the simulations and the observations were calculated and illustrated in a Taylor diagram( Taylor, 2001 ). Considering the skill of models for temperature is better than for precipitation (e.g., Hu et al., 2014 ; Jiang et al., 2016, 2020a ),three thresholds were set to choose relatively good models based only on the ability of models for Xinjiang precipitation. They were, the SCC was statistically significant at the 0.01 level, the RDS was less than 2.5,and the CRMSE was less than 2.0, with regard to the annual precipitation. In total, 29 models were selected as good models, which were also evaluated for both temperature and precipitation. In addition, the results from the arithmetic mean of multiple models and their median were also compared.

3. Results

3.1. Observed and simulated temperature climatology

The geographical distribution of climatological annual temperature in Xinjiang is consistent with the unique topographic features of “Three Mountains and Two Basins ”( Fig. 1 (c)). The observed pattern can be captured well by the medians of the 42 models ( Fig. 1 (a)) and the 29 selected models ( Fig. 1 (b)), characterized by a meridional elevationrelated distribution. Model—observation discrepancies also exist. Compared to the observations, the multi-model median has warm biases over most regions with a maximum of 4.9°C, but cold biases over the Kunlun Mountains in southern Xinjiang with a maximum level of ? 6.9°C( Fig. 1 (d)). On average, the simulated average temperature in Xinjiang is 0.1°C higher than the observation. Such a level of bias is smaller than the ? 1.2°C in CMIP5 ( Wang et al., 2020b ), indicating an improved ability from CMIP5 to CMIP6.

Fig 1. Climatological annual and seasonal temperature (units: °C) over Xinjiang during the period 1995—2014 as obtained from the median of 42 CMIP6 models(left-hand column), the median of 29 selected good models (second column), the observation of CN05.1 (third column), and the difference between the 42-model median and the observation (right-hand column), respectively. The regional average value in Xinjiang is provided in the top left in each panel (units: °C).

Fig 2. Taylor diagram for climatological annual and seasonal temperature over Xinjiang between the CMIP6 models and the observation of CN05.1 during the period 1995—2014. Each number represents a model ID as listed in Table 1 . The red asterisk represents the median of 42 CMIP6 models, and the green circle represents their arithmetic mean. The black asterisk and the black circle represent the median and the mean of the 29 selected good models, respectively. The oblique dashed line shows the 99% confidence level.

Fig 3. Climatological annual and seasonal precipitation (units: mm d ? 1 ) over Xinjiang for the period 1995—2014 as obtained from 42-model median (left-hand column), the 29-model median (second column) of CMIP6, the observation of CN05.1 (third column), and the difference between the median of 42 models and the observation (right-hand column), respectively. The regional average value in Xinjiang is provided in the top left in each panel (units: mm d ? 1 ).

Quantitatively, as shown in Fig. 2 , the SCCs of the 42 models between the simulated and observed climatological annual temperature vary from 0.95 to 0.98, and hence the observational geographical distribution is simulated well by the models. Meanwhile, the RSDs are 1.05—1.48, and the CRMSEs are 0.22—0.58. That means that the models can reasonably simulate the spatial variability of climatological annual temperature in Xinjiang. In comparison, the SCC, RSD, and CRMSE are 0.97, 1.21, and 0.34 for the multi-model mean, and 0.97, 1.19, and 0.32 for the multi-model median, respectively. This indicates that both the arithmetic mean and median of the 42 models outperform most individual models, as previous studies have found in various regions ( Xu and Xu, 2012 ; Hu et al., 2014 ; Zhao et al., 2014 ; Jiang et al., 2016, 2020a ;Lovino et al., 2021 ). In addition, we further evaluate the 29 selected good models. The SCC, RSD, and CRMSE of the multi-model mean are 0.97, 1.20, and 0.33, and those of the multi-model median are 0.97,1.16, and 0.30, respectively. Both the median and arithmetic mean of the 29 good models share similar statistics to those of the 42 models.Comparably, the SCC, RSD, and CRMSE of the median of the 49 CMIP5 models are 0.89, 0.76, and 0.50, respectively ( Jiang et al., 2020a ), indicating that the CMIP6 models perform better in reproducing the temperature in Xinjiang than their CMIP5 counterparts.

For the seasons, the observational geographical distribution of temperature is characterized by high values in the basins and low values in the mountains —similar to the climatological annual temperature( Fig. 1 (g, k, o, s)). The regional average temperature is 19.2°C and has a maximum of approximately 30.0°C in summer ( Fig. 1 (k)), while it is? 10.5°C and has a minimum of near ? 28.0°C in winter ( Fig. 1 (s)). This indicates that an obvious seasonality of temperature exists in Xinjiang.The multi-model median also reasonably captures the observed climatological seasonal temperature features in Xinjiang, with little difference between the 42 and 29 models ( Fig. 1 (e—t)). It should be noted that the simulated climatological temperature in both the annual and seasonal means always exhibits colder biases at the steep mountain edge, as seen in earlier generations of CMIP models ( Xu and Xu, 2012 ; Chen and Frauenfeld, 2014 ; Jiang et al., 2016 ). Averaged over Xinjiang, the differences between the simulated and observed temperature are ? 1.6 °C in spring, 0.5 °C in summer, ? 0.2°C in autumn, and 1.3°C in winter( Fig. 1 (h, l, p, t)).

Similar to the annual temperature, both the arithmetic mean and median of the 42 models outperform most individual models ( Fig. 2 ).The SCCs between the 42 models and the observations vary from 0.95 to 0.99 in spring, summer, and autumn, and from 0.78 to 0.90 in winter. The RSDs of the multi-model mean are 1.16, 1.18, 1.22, and 1.20 in spring, summer, autumn, and winter, respectively. The CRMSEs of the multi-model mean are 0.28 in spring, 0.25 in summer, 0.31 in autumn, and 0.61 in winter. On the whole, the models are better in spring,summer, and autumn than in winter. In addition, the SCCs, RSDs, and CRMSEs for the 42-model median are 0.98, 1.14, and 0.25 in spring,0.99, 1.16, and 0.23 in summer, 0.98, 1.20, and 0.29 in autumn, and 0.86, 1.16, and 0.60 in winter, respectively; and for the median of the 29 good models, they are 0.98, 1.13, and 0.24 in spring, 0.99, 1.15,and 0.23 in summer, 0.98, 1.18, and 0.29 in autumn, and 0.86, 1.16,and 0.60 in winter. This shows that the median of the 29 good models have similar abilities to the 42 models and outperform most individual models. The same conclusion holds for the arithmetic mean.

3.2. Observed and simulated precipitation climatology

Fig. 3 (a—d) shows the climatological annual precipitation of Xinjiang in the models, observations, and their differences. The precipitation is spatially inhomogeneous in this region, with more in the mountains (maximum: 1.4 mm d?1) and less in basins (minimum: 0.1 mm d?1) ( Fig. 3 (c)). The 42 and 29 selected CMIP6 models reasonably simulate the observational spatial distribution of annual precipitation in Xinjiang ( Fig. 3 (a, b)). On the other hand, the models have limitations to exactly reproduce the precipitation distributions in subregions of Xinjiang ( Fig. 3 (d)). An overestimate within 430% occurs over mountainous areas, but an underestimate within ? 57% exists over basins.Wu et al. (2011) suggested that the model error itself, the observation data error, and the insufficient and uneven distribution stations in Xinjiang all have led to those biases. On average, the regional precipitation in Xinjiang is overestimated by 89% in CMIP6 models, which is lower than the 94% in CMIP5 models ( Wang et al., 2020b ). Such improvement in the CMIP6 models can also be seen in previous assessments focusing on the whole of China ( Jiang et al., 2020a ; Zhou, 2020 ; Zhang et al.,2021b ).

Quantitative analysis shows that the skill of the 42 models for annual precipitation varies by model ( Fig. 4 ). The SCCs of five models(namely, BCC-ESM1, CESM2-FV2, MCM-UA-1-0, MIROC-ES2L, and Nor-CPM1) do not exceed the 99% confidence level, indicating that these models cannot simulate the spatial distribution of annual precipitation in Xinjiang well. The RSDs range from 0.17 to 2.83, and those of three models (namely, CNRM-CM6-1, CNRM-ESM2-1, and MRI-ESM2-0) are larger than 2.5, indicating that they greatly overestimate the spatial variability of the annual precipitation. In addition, the CRMSEs from 10 models (namely, BCC-ESM1, CNRM-CM6-1, CNRM-ESM2-1, GISS-E2-1-G-CC, GISS-E2-1-G, GISS-E2-1-H, HadGEM3-GC31-LL, MIROC-ES2L,MRI-ESM2-0, and UKESM1-0-LL) are larger than 2.0, and hence they are relatively poor at simulating the climatology of annual precipitation in Xinjiang. In comparison, the SCC, RSD, and CRMSE for the 42-model median are 0.49, 1.54, and 1.36, while they are 0.52, 1.48, and 1.29 for the arithmetic mean, respectively. This indicates that both the multimodel median and arithmetic mean outperform most individual models. In addition, the arithmetic mean and median of the 29 good models have similar skills to those of the 42 models. For example, the SCC, RSD,and CRMSE of the 29-model median are 0.55, 1.49, and 1.26, respectively. Similar to the temperature, the CMIP6 models are better than the CMIP5 models in reproducing the precipitation in Xinjiang, since the associated statistics of the median of 49 CMIP5 models are 0.39 for SCC,1.73 for RSD, and 1.63 for CRMSE ( Jiang et al., 2020a ).

The observational seasonal precipitation shares a similar geographical distribution to the annual mean at the large scale, and the summer receives the largest precipitation among the seasons in Xinjiang ( Fig. 3 (g,k, o, s)). Overall, the CMIP6 models simulate the spatial pattern of seasonal precipitation reliably, with their skills varying with season ( Fig. 3 ).The models have a relatively large regionally averaged bias of 256%in spring, which is mainly caused by the overestimation in the Kunlun Mountains ( Fig. 3 (h)). In summer, the models have the smallest average bias of ? 3%, and this is mainly because the overestimated precipitation in Kunlun is offset by underestimation in most other areas ( Fig. 3 (l)).The regionally averaged difference between simulation and observation is 84% in autumn and 258% in winter, although underestimation exists in northern Xinjiang. It should be noted that the overestimated precipitation in Kunlun exists in both the annual and seasonal means, which has also been seen in previous model assessments over China within the framework of CMIP6 ( Jiang et al., 2020a ) and earlier generations ( Jiang et al., 2016 ).

In summer, the SCCs of the 42 models range from ? 0.06 to 0.80,the RSDs are 0.12—3.32, and the CRMSEs vary from 0.86 to 2.75. Meanwhile, the SCC, RSD, and CRMSE are 0.62, 1.07, and 0.90 for the multimodel median, and 0.65, 1.08, and 0.88 for the multi-model mean, respectively. This indicates that the arithmetic mean and median of the 42 CMIP6 models outperform most individual models ( Fig. 4 ). Similar improvement also occurs in spring, autumn, and winter. In addition,the SCC, RSD, and CRMSE of the 42-model median are 0.30, 2.24, and 2.17 in spring, 0.61, 1.24, and 1.01 in autumn, and 0.51, 1.47, and 1.29 in winter, respectively. This indicates that the models perform better in summer than in autumn, and the worst in spring. A further analysis of the 29 good models shows their arithmetic mean and median are comparable between each other, and have similar statistics to the aforementioned 42 models on the seasonal scale.

4. Conclusions

This study evaluated the ability of 42 CMIP6 models in simulating the annual and seasonal temperature and precipitation climatology over Xinjiang during the period 1995—2014. The primary conclusions are as follows:

(1) The climatological annual and seasonal temperatures are reasonably reproduced by CMIP6 models, and simulations match well the unique topographic features of “Three Mountains and Two Basins ” in Xinjiang. Compared to the observation, overestimation mainly occurs in basins while underestimation is shown in mountain areas. The regional average bias is 0.1°C for the annual mean, ? 1.6°C in spring, 0.5°C in summer, ? 0.2°C in autumn, and 1.3°C in winter.

(2) Most models can capture the spatial characteristics of annual and seasonal precipitation in Xinjiang, with more in mountains and less in basins. Models generally overestimate the annual and seasonal (except summer) precipitation, especially in the Kunlun Mountains. The regionally averaged bias is 89% for the annual mean, 256% in spring, ? 3% in summer, 84% in autumn, and 258% in winter.

(3) The 42 models show larger discrepancies in simulating precipitation than temperature. For the seasons, the models perform the best for both variables in summer and the worst for temperature in winter and for precipitation in spring. Both the arithmetic mean and median of the 29 good models outperform most individual models and have similar skills to those of the 42 models.

Funding

This research was supported by the National Natural Science Foundation of China [grant number 41991284].

Acknowledgements

We sincerely thank the two anonymous reviewers for their insightful and constructive comments, which certainly helped to improve this manuscript. We also acknowledge the climate modeling groups (listed in Table 1 ) for producing and sharing their model output.