Skill improvement of the yearly updated reforecasts in ECMWF S2S prediction from 2016 to 2022

Yiho Peng ,Xiolei Liu ,Jingzhi Sub,,? ,Xinli Liu ,Yixu Zhng

a State Key Laboratory of Severe Weather,Chinese Academy of Meteorological Sciences,Beijing,China

b Center for Earth System Modeling and Prediction of CMA(CEMC),Beijing,China

c Chengdu University of Information Technology,Chengdu,China

Keywords: Reforecast S2S Prediction skill ECMWF

ABSTRACT Hazardous weather events are often accompanied by subseasonal processes,but the forecast skills of subseasonal prediction are still limited.To assess the skill improvement of the constantly updated model version in ECMWF subseasonal-seasonal (S2S) prediction from 2016 to 2022,the performance of yearly updated reforecasts was evaluated against ERA5 reanalysis data using the temporal anomaly correlation coefficient (TCC) as a metric.The newly updated reforecasts exhibit stable superiority at the weather scale of the first two weeks,regardless of whether the 2-m temperature or precipitation forecast is being considered.At the subseasonal time scale starting from the third week,some slight improvements in prediction skills are only found in several tropical regions.Generally,the week-3 TCC values averaged over global land grids still reflect an advancement in prediction skills for updated reforecasts.For the Madden—Julian Oscillation (MJO),reforecasts can reproduce the characteristics of eastward propagation,but there are deviations in the intensity and propagation range of convection anomalies for reforecasts of all seven years.Based on an evaluation of MJO prediction skill using the bivariate anomaly correlation coefficient and bivariate root-mean-square error,some differences are apparent in the MJO prediction skills among the updated reforecasts,but the improvements do not increase monotonically year by year.Despite the inherent limitation of S2S prediction,positive progress has already been achieved via the constantly updated S2S prediction in ECMWF,which reinforces the confidence in further collaboratively improving S2S prediction in the future.

1.Introduction

After more than half a century of continuous development,weather forecasting and climate prediction have made great progress.However,there is a clear prediction gap between the subseasonal to seasonal(S2S)scale ranges(from two weeks to a whole season).The weather processes that cause severe disasters are often accompanied by anomalous intensity and stable persistence of atmospheric circulation systems,with subseasonal scale characteristics (Zhai et al.,2013).In order to reduce the huge economic and social losses caused by meteorological disasters,research on S2S scale prediction has become a key issue in the field of atmospheric research.

There are inherent difficulties for S2S prediction.On the one hand,the information contained in the initial state of the atmosphere is gradually lost due to the effect of atmospheric chaos when calculated in numerical weather models (Lorenz,1969).Also,the evolution of other earth system components (e.g.,ocean,sea ice,land surface) is usually not synchronously simulated,making the traditional numerical weather forecast have an upper limit of two weeks.On the other hand,climate models incorporate the coupling effects between various subsystems of the earth system.These earth system components,which interact with the atmosphere,provide boundary conditions to guide the evolution of the atmosphere.As traditional climate models are computationally more expensive than weather models,climate models have to adopt a lower grid resolution and less detailed representation of atmospheric processes.For S2S prediction,both the initial value problem and the boundary problem are important,so there are many more difficulties in obtaining accurate S2S forecasts.

The international S2S Prediction Project was jointly created by the World Weather Research Programme and the World Climate Research Programme in awareness of the necessity to gather various communities together to fill the knowledge gaps in S2S prediction(Mariotti et al.,2018).The product set of the S2S Prediction Project,a collection of forecast products from 11 major forecast centers around the world,provides the possibility to systematically study S2S prediction skills.Among all the models issued by the 11 institutions,the model from the European Centre for Medium-Range Weather Forecasts (ECMWF) has the highest forecast skill,the slowest decline/rise in anomaly correlation coeffi-cient (ACC)/root-mean-square error (RMSE) with prediction time,and the longest prediction duration of any model,at up to 32 days(Vitart and Robertson,2018;Peng et al.,2021).

To improve S2S prediction skills,ECMWF updates model versions almost every year (Table S1),with improved initial conditions,physical parameterizations,and refined horizontal resolution (Table S2).The S2S prediction in ECMWF is arranged with on-the-fly configuration,in which all the reforecasts are produced at the same time as the real-time forecasts.Another S2S configuration is based on fixed reforecasts,in which all the reforecasts for all past dates are produced only once during the lifetime of a given model version.Comparatively,the on-the-fly configuration is far more computationally expensive in terms of achieving the updated reforecasts in real-time.In other words,one more new reforecast must be operationally constructed every year.

In light of the above,some questions need to be addressed.Is the degree of improvement in S2S prediction skills obtained by the on-the-fly configuration cost-effective? Furthermore,are there necessarily better prediction skills with the new version reforecasts updated continuously,and where and which features have been improved?Particularly,more attention should be paid to the Madden—Julian Oscillation(MJO),as it is a primary source of predictability at the S2S scale(Ding and Liang,2010;Neena et al.,2014;Waliser et al.,1999).Are there any advances in the forecast skill for the MJO? The purpose of this paper is to explore the questions mentioned above.

2.Data and methods

2.1.Data

2.1.1.Observational datasets

ERA5 is the fifth generation ECMWF atmospheric reanalysis of the global climate.Reanalyses assemble model data with observations from across the world into a global dataset (Bell et al.,2021).This study used daily averaged data from ERA5 as observations released by the Copernicus Climate Change Service (C3S),with a horizontal resolution of 1° × 1°.The daily variables used included air temperature at 2 m,zonal wind at 200 hPa and 850 hPa,sea surface temperature(SST),top net longwave radiation flux,and total precipitation.

2.1.2.Model outputs

The ECMWF model simulates initial conditions with singular vectors and ensembles of data assimilation (Richter et al.,2022).The ECMWF reforecast includes one control forecast member and 10 perturbation forecast members.The reforecasts of these 11 members are calculated for the ensemble average.This study used daily averaged data of all ECMWF S2S model versions from 2016 to 2022.The 20-year reforecasts were carried out every Monday and Thursday,with a forecasting time range of 46 days.The variables mentioned above were also adopted in the reforecast data,and the resolution was 1.5°×1.5°.

2.2.Method

The model version dates used were from 2016 to 2022,each with a corresponding 20-year reforecast dataset.A common hindcast period of 2002—2015 was covered by all the model version dates,and this period was selected to compare the differences among the model versions.The observational anomaly fields were obtained by removing the climatological annual cycle based on a reference period of 2002—2015.The reforecast anomalies were calculated by removing the model reforecasts’climatology and the previous 120 days’time mean anomalies as smoothed data.

This study calculated the weekly averages of the forecast period from the ensemble mean of model members.The skill of the temporal anomaly correlation coefficient (TCC) was calculated for temporal anomalies at each grid point(Wilks,1995).From the TCC,information about how well the variability of the forecasted anomalies matched the observed variability could be achieved.The TCC was calculated between the predicted and observed anomalies as follows:

wherea(t)denotes the verification anomaly values at timet,b(t,τ)denotes the respective forecasts for timetwith a lead time ofτdays,ˉaandˉbare the mean values of observed and predicted anomalies,respectively,andNis the number of predictions.

A common measurement of MJO prediction skill makes use of two Real-time Multivariate MJO (RMM) indices (Wheeler and Hendon,2004).The anomalies of outgoing longwave radiation (OLR) and zonal wind at 200 and 850 hPa are then projected onto two observed principal components of the combined empirical orthogonal functions(EOFs) to achieve the RMM indices (RMM1 and RMM2).The bivariate ACC and bivariate RMSE between the observed and predicted RMM indices were used here to measure the forecast skill of the numerical model as a function of forecast lead times (Lin et al.,2008).The ACC and RMSE were calculated between the observed and predicted RMM indices as follows:

wherea1(t)anda2(t)are the verification RMM1 and RMM2 at timet,b1(t,τ)andb2(t,τ)are the respective forecasts RMM1 and RMM2 for timetwith a lead time ofτdays.

Week 3 is beyond the weather time scale,and predictability due to atmospheric initial conditions is largely absent (Lorenz,1965).Also,predictability due to slower varying components of the climate system present in the initial anomaly will change little over a 3-week forecast.Therefore,skill due to these mechanisms would be present in a persistence forecast.Hence,some week-3 mean features will be examined to represent the S2S prediction skill(Pegion et al.,2019).

3.Results

3.1.SST,2-m temperature,and precipitation prediction skill

High correlations between the forecasted and observed SST exist over almost the whole globe,with TCCs above 0.5 over the majority of ocean areas and TCCs above 0.8 in the tropical Pacific Ocean in the third week (Fig.S1).Lower TCC values (<0.5) can be found over the regions south of 45°S,and some regions corresponding to the strong western currents,such as the Gulf Stream and Kuroshio extension.Compared with the 2016 reforecast,there are apparent TCC increments in the reforecasts since 2017(Fig.S1(b—g)).Significant TCC increments of reforecasts of 2017—2022 are located in these regions:south of 45°S,the northern Atlantic,and the sea of Okhotsk.Furthermore,the 2019—2022 reforecasts also have other increments of correlation coefficients over the northern side of New Guinea Island and western side of the South African plateau,and there is no significant difference between the forecasts of these four years.The TCCs of reforecasts of 2022 show a relatively obvious increase in the eastern region of the Malay Archipelago.

For 2-m temperature,TCC values in the tropics (>0.4) are higher than in the middle and high latitudes(～0.3)in general(Fig.1(a)).From the third week,on the subseasonal time scale,in terms of increments from 2016,the improvement of forecasts from 2017 and 2018 in most regions is not significant.On the contrary,the TCCs of reforecasts in 2018 indicate decreases(～0.05)in the North American and eastern European plains (Fig.1(c)).The TCC values from 2019 to 2022 have increments(>0.05) mainly in the tropical regions (Fig.1(d—g)),such as Southwest Asia,central Africa,and South America,and particularly,TCC increments above 0.1 can be found over the Azande plateau in central Africa.In South America,TCC increments of reforecasts of 2019 and 2021 are more obvious than for other years,especially for the reforecasts of 2021(Fig.1(f)).

Fig.1.(a) TCC of 2-m temperature for reforecasts in 2016 over land for week 3 (average of forecast days 15—21).Panels (b—g) show the TCC differences between 2017—2022 and 2016.The calculation was performed over reforecasts during the period 2002—2015.

The prediction skill for 2-m temperature shows a constant increase as the reforecasts are built from year to year(Fig.2(a)).In the first two weeks,within the weather-scale time range,TCC values of all seven reforecast years reach around 0.86 in the first week and 0.6 in the second week,and increase year by year.From 2019 to 2021,the globally averaged TCC value steadily improves during the third week (from 0.36 to 0.39)and fourth week(from 0.24 to 0.25).The newer the reforecast year,the better the prediction skill tends to be.However,for the reforecasts of the most recent year (2022),the TCC drops to 0.37 in the third week but ranks first at 0.26 in the fourth week.By comparison,the 2018 reforecast performs relatively poorly during these two weeks.For the fifth and sixth week,the TCC values of all the reforecasts drop below 0.2.From then,all the prediction skills are similar to each other,and the new reforecasts during these years show no obvious advantages.

Fig.2.(a) TCC of 2-m temperature averaged over global land grids for reforecasts of 14 years (2002—2015) for each year during 2016—2022.The TCC calculation was carried out based on the ensemble mean of 11 members for each reforecast,and then averaged over the global land grids weighted by grid area.(b) As in (a)but for precipitation.

The prediction skill of precipitation is generally lower compared with the other variables.Similar to 2-m temperature,the yearly trends of TCC values for precipitation show a characteristic gradual increase in the first week (from 0.47 to 0.51) and the second week (from 0.24 to 0.27) (Fig.2(b)).This too reflects the improving prediction skill of the newly updated reforecasts at the weather scale.The precipitation TCC during the third week increases slightly from 2016 to 2022 (from 0.11 to 0.12).There are almost no improvements in the precipitation TCC values during the fifth and sixth weeks.

The precipitation prediction skill is slightly higher(>0.3)in the tropics,such as in northern South America,a small area in the east-central part of the African continent,and the Malaysian Islands (Fig.3(a)).For 2017—2022,there are positive increments located over the Indian Peninsula(>0.15)and southern Africa(>0.05),but the TCC increments from 2016 are still smaller than 0.1 at almost all global land grid points(Fig.3(b—g)).This is because precipitation is generally hard to predict on subseasonal time scales,in addition to some areas mentioned above.

Fig.3.(a) TCC of precipitation for reforecasts in 2016 over land for week 3 (average of forecast days 15—21).Panels (b—g) show the TCC differences between 2017—2022 and 2016.The calculation was performed over reforecasts during the period 2002—2015.

3.2.MJO prediction skill

In the 1970s,the MJO was discovered,and this tropical lowfrequency oscillation is the dominant mode of intraseasonal variation in the tropical atmosphere(Madden and Julian,1972).The MJO has an important influence on monsoon outbreaks,the development of ENSO events,and tropical cyclone generation(Jeong et al.,2008;Kessler and Kleeman,2000;Fu and Hsu,2011).In addition to its important influence on weather and climate at low latitudes,the MJO can also modify the meridional circulation and further influence the circulation and precipitation at middle and high latitudes by exciting remotely correlated wave trains (Cassou,2008).Therefore,accurate prediction of the MJO is of great value as it helps to expand subseasonal prediction skills globally,improves the response capability in disaster prevention and mitigation,and makes an important contribution to the normal operation of society.

The MJO’s evolution can be well represented by the OLR anomalies,representing the convection anomalies associated with the MJO.Anomalies of OLR,and 850 hPa and 200 hPa zonal winds,were projected onto the two observed leading multivariate EOF modes to obtain the RMM indices(Wheeler and Hendon,2004).The first(second)mode represents the phases of anomalous convection in the Indian Ocean(western Pacific).The RMM1 and RMM2 indices were used as the horizontal and vertical coordinates,respectively,to form a 2D spatial phase diagram.The phase diagram divides 0°to 360°into eight spatial phases,which were used to characterize the location of the main convection zone of the MJO.Then,the composite OLR of propagating MJO events in each phase in the third week(days 15—21)of the start of each MJO event was calculated based on the model reforecasted data for each year.Convection anomalies propagate eastwards from the Indian Ocean(around 75°E)to the Pacific(Fig.4(a)).The results of the reforecasts for all seven years were able to reproduce the characteristics of this eastern propagation in week 3(Fig.4(b—g)).However,the reforecasts of the intensity and propagation range of the OLR anomalies are biased at the subseasonal time scale.All seven years’reforecasts have higher values for the intensity of OLR anomalies.The reforecast results for 2021 are relatively closer to the observed values in comparison.

Fig.4.Week 3 (average of days 15—21) composite OLR (W m-2) averaged over 10°S—10°N as a function of longitude (x-axis) and phase (y-axis) for MJO events based on RMM index amplitude ≥1 in winter seasons (November to April) during 2002—2015.The composite OLR of MJO propagation for each phase in the third week(days 15—21)of the start of each MJO event was calculated based on(a)ERA5 reanalysis data and(b—h)model reforecasts for each year.

The MJO prediction skill during boreal wintertime (November to April),as amplitude and phase error,was measured with the ACC and RMSE for reforecasts updated by year.Determined by the maximum lead time with the 11-member ensemble mean ACC exceeding 0.5,the ECMWF reforecasts of 14 years from 2016 to 2022 are all above 28 days (Fig.5(a)).The ACC for reforecasts of 2020 is below 0.5 by 28 days,while reforecasts of 2017 have the longest maximum lead time(31 days).Meanwhile,the latest reforecasts(for 2022)have a lead time of 29 days,ranking third in the data of the seven years.Influenced by the reforecasted data update,the change in the maximum lead time is between 1 and 4 days,which cannot be considered as a small amount for subseasonal prediction.However,the updated reforecasts do not directly lead to a monotonic improvement in prediction skills;otherwise,it would basically show a trend that the maximum lead time extends with the update of the reforecasts.

The trend for the RMSE of reforecasts from 2016 to 2022 is basically almost the same(Fig.5(b)).With RMSE=as the boundary,the maximum lead time of each year is 36—39 days.The reforecasts of 2021 have the longest maximum lead time of 39 days,followed by those of 2017(38 days),while the other five years’maximum lead times are around 36 days overall.Updated reforecasts do not show improving prediction skills.Like the analysis of the ACC,at the subseasonal time scale,improvements in MJO prediction skills are not greatly affected by updates of reforecasts.

4.Discussion and conclusion

To improve S2S prediction skills,ECMWF periodically updates its model versions,with an on-the-fly configuration.By comparing with ERA5 reanalysis data,some steady advances can be found in the prediction skill from year to year,particularly in the first two weeks.In the first and second week,which is within/around the weather timescale,the TCC of 2-m temperature and precipitation is increasing year by year as the reforecasts are updated.On this time scale,investment in on-the-fly configuration seems effective for prediction skill improvement.

From week 3,at the subseasonal time scale,some enhancements in prediction accuracy can be found in the updated reforecasts.The globally averaged TCC of 2-m temperature during weeks 3 and 4 has steadily improved since 2019,and such advances have mainly occurred in the tropics,such as in Southwest Asia,central Africa,and South America.The precipitation TCC was found to have slightly increased from 2016 to 2022,with positive increments located over the Indian Peninsula and southern Africa.There is little improvement in the precipitation TCC values during weeks 5 and 6.

For the MJO prediction,the results of the reforecasts for all seven years were able to reproduce the characteristics of its eastwards propagation;however,it was difficult for the reforecasts to capture the intensity and propagation range of the OLR anomalies at the subseasonal time scale.Assessment of the MJO prediction skill with the ACC and RMSE did not show any obvious features of the maximum lead time extending monotonically with the updates of reforecasts.

To check why the MJO prediction skill has not increased year by year,the prediction skill of OLR and the zonal winds at 850 hPa and 200 hPa were also calculated.It is usually considered that the OLR and precipitation locally correspond to each other well in tropical convective areas.From the spatial distribution of the TCCs for precipitation and OLR in the third week (Fig.3 and Fig.S2),there are many obvious mismatches between them,such as over the Indian Peninsula.Such mismatches of prediction skill between OLR and precipitation may be related to the simulation/forecasting performance of the numerical model.Also,for the zonal winds at 850 hPa and 200 hPa,there are no obvious improvements among the updated reforecasts(Figs.S3—S5).As a result,the MJO prediction skill failed to show an increasing trend year by year.

In general,for subseasonal scale prediction,which is reflected in the prediction of the MJO,the updated reforecasts have the ability to demonstrate the main features of the MJO.At the same time,there is still room for improvement in accuracy.Updates of reforecasts will affect the MJO prediction skill;however,they do not directly lead to a monotonic improvement.From very early on,ECMWF has invested a lot of resources in S2S prediction.However,due to many objective scientific problems that are currently difficult to overcome in subseasonal prediction,even though the reforecasts are updated constantly,obvious improvements in MJO prediction skill are rarely found.

Due to the gradual loss of initial condition information and the incomplete description of the earth system,S2S prediction remains a great challenge for numerical model prediction.Many research institutes and organizations have already made considerable progress in S2S prediction.Through continuous efforts to polish its S2S prediction system,ECMWF has provided updated model versions from year to year.Such updated reforecasts have definitely achieved positive improvements in S2S prediction.This encourages more joint contributions between scientific communities in different fields for continuous progress in S2S prediction.

Funding

This work was supported by the National Key Research and Development Program [grant numbers 2019YFC1510002 and 2022YFC3004203].

Acknowledgments

The comments from the two anonymous reviewers provided valuable suggestions that helped improve the paper.

Supplementary materials

Supplementary material associated with this article can be found,in the online version,at doi:10.1016/j.aosl.2023.100357.