Benchmarking CMIP5 models with a subset of ESA CCI Phase 2 data using the ESMValTool

The Coupled Model Intercomparison Project (CMIP) is now moving into its sixth phase and aims at a more routine evaluation of the models as soon as the model output is published to the Earth System Grid Federation (ESGF). To meet this goal the Earth System Model Evaluation Tool (ESMValTool), a community diagnostics and performance metrics tool for the systematic evaluation of Earth system models (ESMs) in CMIP, has been developed and a first version (1.0) released as open source software in 2015. Here, an enhanced version of the ESMValTool is presented that exploits a subset of Essential Climate Variables (ECVs) from the European Space Agency's Climate Change Initiative (ESA CCI) Phase 2 and this version is used to demonstrate the value of the data for model evaluation. This subset includes consistent, long-term time series of ECVs obtained from harmonized, reprocessed products from different satellite instruments for sea surface temperature, sea ice, cloud, soil moisture, land cover, aerosol, ozone, and greenhouse gases. The ESA CCI data allow extending the calculation of performance metrics as summary statistics for some variables and add an important alternative data set in other cases where observations are already available. The provision of uncertainty estimates on a per grid basis for the ESA CCI data sets is used in a new extended version of the Taylor diagram and provides important additional information for a more objective evaluation of the models. In our analysis we place a specific focus on the comparability of model and satellite data both in time and space. The ESA CCI data are well suited for an evaluation of results from global climate models across ESM compartments as well as an analysis of long-term trends, variability and change in the context of a changing climate. The enhanced version of the ESMValTool is released as open source software and ready to support routine model evaluation in CMIP6 and at individual modeling centers.

[1]  W. Collins,et al.  Evaluation of climate models , 2013 .

[2]  Victor Brovkin,et al.  Global biogeophysical interactions between forest and climate , 2009 .

[3]  D. E. Harrison,et al.  Implementation Plan for the Global Observing System for Climate in Support of the UNFCCC (2010 Update) , 2010 .

[4]  Yi Y. Liu,et al.  Trend-preserving blending of passive and active microwave soil moisture retrievals , 2012 .

[5]  Justus Notholt,et al.  The Total Carbon Column Observing Network , 2011, Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences.

[6]  Martin Dameris,et al.  Technical Note: A new global database of trace gases and aerosols from multiple sources of high vertical resolution measurements , 2008 .

[7]  M. Collins,et al.  The internal climate variability of HadCM3, a version of the Hadley Centre coupled model without flux adjustments , 2001 .

[8]  Yi Y. Liu,et al.  Evaluating global trends (1988–2010) in harmonized multi‐satellite surface soil moisture , 2012 .

[9]  S. Kern,et al.  Inter-comparison and evaluation of sea ice algorithms: towards further identification of challenges and optimal approach using passive microwave observations , 2015 .

[10]  Alain Hauchecorne,et al.  Harmonized dataset of ozone profiles from satellite limb and occultation measurements , 2013 .

[11]  L. Remer,et al.  The Collection 6 MODIS aerosol products over land and ocean , 2013 .

[12]  Veronika Eyring,et al.  Evolving Obs4MIPs to Support Phase 6 of the Coupled Model Intercomparison Project (CMIP6) , 2015 .

[13]  Yi Y. Liu,et al.  Developing an improved soil moisture dataset by blending passive and active microwave satellite-based retrievals , 2011 .

[14]  H. Eskes,et al.  Indicators of Antarctic ozone depletion , 2005 .

[15]  Judith A. Curry,et al.  Sea Ice-Albedo Climate Feedback Mechanism , 1995 .

[16]  W. Collins,et al.  The Community Earth System Model: A Framework for Collaborative Research , 2013 .

[17]  Wouter Dorigo,et al.  Potential and limitations of multidecadal satellite soil moisture observations for selected climate model evaluation studies , 2013 .

[18]  Josefino C. Comiso,et al.  Variability and Trends in Antarctic Surface Temperatures from In Situ and Satellite Infrared Measurements , 2000 .

[19]  Luis Kornblueh,et al.  Sensitivity of Simulated Climate to Horizontal and Vertical Resolution in the ECHAM5 Atmosphere Model , 2006 .

[20]  J. Randerson,et al.  Causes and implications of persistent atmospheric carbon dioxide biases in Earth System Models , 2013 .

[21]  Urs Wegmüller,et al.  Multi-temporal Synthetic Aperture Radar Metrics Applied to Map Open Water Bodies , 2014, IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing.

[22]  Richard Siddans,et al.  Round-robin evaluation of nadir ozone profile retrievals: methodology and application to MetOp-A GOME-2 , 2014, Atmospheric Measurement Techniques.

[23]  Robert M. Parinussa,et al.  Error Estimates for Near-Real-Time Satellite Soil Moisture as Derived From the Land Parameter Retrieval Model , 2011, IEEE Geoscience and Remote Sensing Letters.

[24]  Brian J. Kerridge,et al.  Tropospheric ozone and ozone profiles retrieved from GOME-2 and their validation , 2014 .

[25]  H. Hasumi,et al.  Improved Climate Simulation by MIROC5: Mean States, Variability, and Climate Sensitivity , 2010, Journal of Climate.

[26]  Hartmut Boesch,et al.  The greenhouse gas project of Esa's climate change initiative (GHG-CCI) : Overview, achievements and future plans , 2015, ATMOS 2015.

[27]  S. Bony,et al.  Climate change projections using the IPSL-CM5 Earth System Model: from CMIP3 to CMIP5 , 2013, Climate Dynamics.

[28]  Dimitris Balis,et al.  Evaluating a new homogeneous total ozone climate data record from GOME/ERS‐2, SCIAMACHY/Envisat, and GOME‐2/MetOp‐A , 2015 .

[29]  V. Canuto,et al.  Present-Day Atmospheric Simulations Using GISS ModelE: Comparison to In Situ, Satellite, and Reanalysis Data , 2006 .

[30]  C. Deser,et al.  Evaluating Modes of Variability in Climate Models , 2014 .

[31]  T. Takemura,et al.  Geoscientific Model Development MIROC-ESM 2010 : model description and basic results of CMIP 5-20 c 3 m experiments , 2011 .

[32]  V. L. Orkin,et al.  Scientific Assessment of Ozone Depletion: 2010 , 2003 .

[33]  Duoying Ji,et al.  Description and basic evaluation of Beijing Normal University Earth System Model (BNU-ESM) version 1 , 2014 .

[34]  Keir Bovis,et al.  Estimating background error covariance parameters and assessing their impact in the OSTIA system , 2016 .

[35]  Zhenya Song,et al.  Development and evaluation of an Earth System Model with surface gravity waves , 2013 .

[36]  M. Herold,et al.  Revisiting land cover observation to address the needs of the climate modeling community , 2011 .

[37]  M. Buchwitz,et al.  SCIAMACHY: Mission Objectives and Measurement Modes , 1999 .

[38]  Simon Read,et al.  ESMValTool (v1.0) – a community diagnostic and performance metrics tool for routine evaluation of Earth system models in CMIP , 2015 .

[39]  Axel Lauer,et al.  Simulating Clouds with Global Climate Models: A Comparison of CMIP5 Results with CMIP3 and Satellite Data , 2013 .

[40]  S. Bony,et al.  The ‘too few, too bright’ tropical low‐cloud problem in CMIP5 models , 2012 .

[41]  Stefan Kern,et al.  The impact of snow depth, snow density and ice density on sea ice thickness retrieval from satellite radar altimetry: results from the ESA-CCI Sea Ice ECV Project Round Robin Exercise , 2015 .

[42]  Toru Nozawa,et al.  MIROC4h—A New High-Resolution Atmosphere-Ocean Coupled General Circulation Model , 2012 .

[43]  Wei Li,et al.  Major forest changes and land cover transitions based on plant functional types derived from the ESA CCI Land Cover product , 2016, Int. J. Appl. Earth Obs. Geoinformation.

[44]  Clive D Rodgers,et al.  Inverse Methods for Atmospheric Sounding: Theory and Practice , 2000 .

[45]  A. Cazenave,et al.  The ESA Climate Change Initiative: Satellite Data Records for Essential Climate Variables , 2013 .

[46]  J. Edmonds,et al.  Scenarios of Greenhouse Gas Emissions and Atmospheric Concentrations , 2007 .

[47]  Antonio Di Gregorio,et al.  Land cover classification system (LCCS): classification concepts and user manual for software version 1.0 , 2000 .

[48]  Hartmut Boesch,et al.  The greenhouse gas climate change initiative (GHG-CCI): Comparative validation of GHG-CCI SCIAMACHY/ENVISAT and TANSO-FTS/GOSAT CO2 and CH4 retrieval algorithm products with measurements from the TCCON , 2013 .

[49]  Veronika Eyring,et al.  Overview of the Coupled Model Intercomparison Project Phase 6 (CMIP6) experimental design and organization , 2015 .

[50]  M. Dameris,et al.  A vertically resolved, monthly mean, ozone database from 1979 to 2100 for constraining global climate model simulations , 2009 .

[51]  Matthias Drusch,et al.  Global Automated Quality Control of In Situ Soil Moisture Data from the International Soil Moisture Network , 2013 .

[52]  Charles Doutriaux,et al.  A More Powerful Reality Test for Climate Models , 2016 .

[53]  W. Wagner,et al.  Evaluation of the ESA CCI soil moisture product using ground-based observations , 2015 .

[54]  C. Jones,et al.  Development and evaluation of an Earth-System model - HadGEM2 , 2011 .

[55]  G. Danabasoglu,et al.  The Community Climate System Model Version 4 , 2011 .

[56]  R. Betts,et al.  Plant functional type classification for earth system models: results from the European Space Agency's Land Cover Climate Change Initiative , 2015 .

[57]  J. Comiso,et al.  Trends in the sea ice cover using enhanced and compatible AMSR‐E, SSM/I, and SMMR data , 2008 .

[58]  Dimitris Balis,et al.  Homogenized total ozone data records from the European sensors GOME/ERS‐2, SCIAMACHY/Envisat, and GOME‐2/MetOp‐A , 2014 .

[59]  A. Sterl,et al.  EC-Earth A Seamless earth-System Prediction Approach in Action , 2010 .

[60]  Yuk L. Yung,et al.  CO2 in the upper troposphere: Influence of stratosphere‐troposphere exchange , 2006 .

[61]  Bronte Tilbrook,et al.  Carbonate chemistry in the Mertz Polynya (East Antarctica): Biological and physical modification of dense water outflows and the export of anthropogenic CO2 , 2014 .

[62]  Min Dong,et al.  The Beijing Climate Center atmospheric general circulation model: description and its performance for the present-day climate , 2009 .

[63]  A. Smirnov,et al.  AERONET-a federated instrument network and data archive for aerosol Characterization , 1998 .

[64]  A. Loew.  Impact of surface heterogeneity on surface soil moisture retrievals from passive microwave data at the regional scale: The Upper Danube case , 2008 .

[65]  Andi Walther,et al.  The Pathfinder Atmospheres–Extended AVHRR Climate Dataset , 2014 .

[66]  Marie-Alice Foujols,et al.  Impact of the LMDZ atmospheric grid configuration on the climate and sensitivity of the IPSL-CM5A coupled model , 2013, Climate Dynamics.

[67]  Yong Xue,et al.  Development, Production and Evaluation of Aerosol Climate Data Records from European Satellite Observations (Aerosol_cci) , 2016, Remote. Sens..

[68]  Tom M. L. Wigley,et al.  Multi-Gas Forcing Stabilization with Minicam , 2006 .

[69]  Masakatsu Nakajima,et al.  Thermal and near infrared sensor for carbon observation Fourier-transform spectrometer on the Greenhouse Gases Observing Satellite for greenhouse gases monitoring. , 2009, Applied optics.

[70]  Jeffrey P. Walker,et al.  Upscaling sparse ground‐based soil moisture observations for the validation of coarse‐resolution satellite soil moisture products , 2012 .

[71]  Karl-Göran Karlsson,et al.  CLARA-A1: a cloud, albedo, and radiation dataset from 28 yr of global AVHRR data , 2013 .

[72]  C. Donlon,et al.  The Operational Sea Surface Temperature and Sea Ice Analysis (OSTIA) system , 2012 .

[73]  Karl E. Taylor,et al.  An overview of CMIP5 and the experiment design , 2012 .

[74]  Amir R. Khoei,et al.  The superconvergence patch recovery technique and data transfer operators in 3D plasticity problems , 2007 .

[75]  Zhi Zong,et al.  A modified superconvergent patch recovery method and its application to large deformation problems , 2004 .

[76]  A. Robock,et al.  The International Soil Moisture Network: a data hosting facility for global in situ soil moisture measurements , 2011 .

[77]  K. Taylor.  Summarizing multiple aspects of model performance in a single diagram , 2001 .

[78]  Peter Bergamaschi,et al.  Atmospheric Chemistry and Physics Atmospheric Methane and Carbon Dioxide from Sciamachy Satellite Data: Initial Comparison with Chemistry and Transport Models , 2022 .

[79]  W. Paul Menzel,et al.  The MODIS cloud products: algorithms and examples from Terra , 2003, IEEE Trans. Geosci. Remote. Sens..

[80]  Scott C. Doney,et al.  Twentieth-Century Oceanic Carbon Uptake and Storage in CESM1(BGC)* , 2013 .

[81]  Ramaswamy,et al.  The dynamical core, physical parameterizations, and basic simulation characteristics of the atmospheric component AM3 of the GFDL global coupled model CM3 , 2011 .

[82]  Corinne Le Quéré,et al.  Carbon and Other Biogeochemical Cycles , 2014 .

[83]  Yanjun Wang,et al.  Spatiotemporal variations of soil moisture in the Tarim River basin, China , 2016, Int. J. Appl. Earth Obs. Geoinformation.

[84]  E. Volodin,et al.  Simulating present-day climate with the INMCM4.0 coupled model of the atmospheric and oceanic general circulations , 2010 .

[85]  R. Kwok,et al.  Variability of Arctic sea ice thickness and volume from CryoSat-2 , 2015, Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences.

[86]  Alexander Loew,et al.  Combined evaluation of MPI‐ESM land surface water and energy fluxes , 2012 .

[87]  Melanie Bräu,et al.  Sea-ice in decadal and long-term simulations with the Max Planck Institute Earth System Model , 2013 .

[88]  W. Wagner,et al.  Fusion of active and passive microwave observations to create an Essential Climate Variable data record on soil moisture , 2012 .

[89]  Alexander Loew,et al.  Evaluation of vegetation cover and land‐surface albedo in MPI‐ESM CMIP5 simulations , 2013 .

[90]  Shi Hu,et al.  Validation and trend analysis of ECV soil moisture data on cropland in North China Plain during 1981-2010 , 2016, Int. J. Appl. Earth Obs. Geoinformation.

[91]  Charles Doutriaux,et al.  Performance metrics for climate models , 2008 .

[92]  J. Edmonds,et al.  Implications of Limiting CO2 Concentrations for Land Use and Energy , 2009, Science.

[93]  R. Lindsay,et al.  Arctic sea ice thickness loss determined using subsurface, aircraft, and satellite observations , 2014 .

[94]  J. Janowiak,et al.  The Version 2 Global Precipitation Climatology Project (GPCP) Monthly Precipitation Analysis (1979-Present) , 2003 .

[95]  Steffen Fritz,et al.  Spatial Accuracy Assessment and Integration of Global Land Cover Datasets , 2015, Remote. Sens..

[96]  Jonah Roberts-Jones,et al.  Daily, Global, High-Resolution SST and Sea Ice Reanalysis for 1985–2007 Using the OSTIA System , 2012 .

[97]  Y. Kerr,et al.  Evaluation of remotely sensed and modelled soil moisture products using global ground-based in situ observations , 2012 .

[98]  Chris D. Jones,et al.  On the significance of atmospheric CO2 growth rate anomalies in 2002–2003 , 2005 .

[99]  Yan Feng,et al.  Improved simulation of Australian climate and ENSO‐related rainfall variability in a global climate model with an interactive aerosol treatment , 2009 .

[100]  Tongwen Wu.  A mass-flux cumulus parameterization scheme for large-scale models: description and test with observations , 2012, Climate Dynamics.

[101]  B. Stevens,et al.  Atmospheric component of the MPI‐M Earth System Model: ECHAM6 , 2013 .

[102]  Robert Ricker,et al.  Sensitivity of CryoSat-2 Arctic sea-ice freeboard and thickness on radar-waveform interpretation , 2014 .

[103]  Veronika Eyring,et al.  Ozone database in support of CMIP5 simulations: results and corresponding radiative forcing , 2011 .

[104]  Alexander Smirnov,et al.  Maritime Aerosol Network as a component of Aerosol Robotic Network , 2009 .

[105]  John P. Burrows,et al.  SCIAMACHY—scanning imaging absorption spectrometer for atmospheric chartography , 1992 .

[106]  R. Reynolds,et al.  The NCEP/NCAR 40-Year Reanalysis Project , 1996, Renewable Energy.

[107]  P. J. Young,et al.  Long‐term ozone changes and associated climate impacts in CMIP5 simulations , 2013 .

[108]  Claire E. Bulgin,et al.  Sea surface temperature datasets for climate applications from Phase 1 of the European Space Agency Climate Change Initiative (SST CCI) , 2014 .

[109]  S. Gualdi,et al.  Effects of Tropical Cyclones on Ocean Heat Transport in a High-Resolution Coupled General Circulation Model , 2011 .

[110]  Jianxiu Qiu,et al.  Comparison of temporal trends from multiple soil moisture data sets and precipitation: The implication of irrigation on regional soil moisture trend , 2016, Int. J. Appl. Earth Obs. Geoinformation.

[111]  Christopher J. Merchant,et al.  Objective Determination of Feature Resolution in Two Sea Surface Temperature Analyses , 2013 .

[112]  E. Kowalczyk,et al.  The ACCESS coupled model: description, control climate and evaluation , 2013 .

[113]  Pierre Friedlingstein,et al.  Uncertainties in CMIP5 Climate Projections due to Carbon Cycle Feedbacks , 2014 .

[114]  J. Thepaut,et al.  The ERA‐Interim reanalysis: configuration and performance of the data assimilation system , 2011 .

[115]  C. Merchant,et al.  A reprocessing for climate of sea surface temperature from the along-track scanning radiometers: Initial validation, accounting for skin and diurnal variability effects , 2012 .

[116]  A. Simmons,et al.  The Concept of Essential Climate Variables in Support of Climate Research, Applications, and Policy , 2014 .

[117]  Christopher J. Merchant,et al.  A reprocessing for climate of sea surface temperature from the along-track scanning radiometers: Basis in radiative transfer , 2012 .

[118]  James M. Russell,et al.  Ground-based assessment of the bias and long-term stability of fourteen limb and occultation ozone profile data records. , 2016, Atmospheric measurement techniques.

[119]  H. Douville,et al.  The CNRM-CM5.1 global climate model: description and basic evaluation , 2013, Climate Dynamics.

[120]  Robert J. D. Spurr,et al.  The GOME-type Total Ozone Essential Climate Variable (GTO-ECV) data record from the ESA Climate Change Initiative , 2015 .

[121]  R. Schnur,et al.  Climate-carbon cycle feedback analysis: Results from the C , 2006 .

[122]  M. Dameris,et al.  Global long-term monitoring of the ozone layer – a prerequisite for predictions , 2009 .

[123]  John M. Haynes,et al.  COSP: Satellite simulation software for model assessment , 2011 .

[124]  William D. Collins,et al.  Evaluation of hydrologic components of community land model 4 and bias identification , 2016, Int. J. Appl. Earth Obs. Geoinformation.

[125]  Michael Buchwitz,et al.  A simple empirical model estimating atmospheric CO 2 background concentrations , 2012 .

[126]  Eric J. Fetzer,et al.  Evaluating CMIP5 models using AIRS tropospheric air temperature and specific humidity climatology , 2013 .

[127]  E. Vermote,et al.  The MODIS Aerosol Algorithm, Products, and Validation , 2005 .

[128]  J. Kay,et al.  The Arctic’s rapidly shrinking sea ice cover: a research synthesis , 2012, Climatic Change.

[129]  J. F. Meirink,et al.  Cloud property datasets retrieved from AVHRR, MODIS, AATSR and MERIS in the framework of the Cloud_cci project , 2017 .

[130]  Duane E. Waliser,et al.  Satellite Observations for CMIP5: The Genesis of Obs4MIPs , 2014 .

[131]  M. Wahlen,et al.  Interannual extremes in the rate of rise of atmospheric carbon dioxide since 1980 , 1995, Nature.

[132]  Gregory R. McGarragh,et al.  The Community Cloud retrieval for CLimate (CC4CL) – Part 1: A framework applied to multiple satellite imaging sensors , 2017 .

[133]  Christopher D. Barnet,et al.  Accuracy of geophysical parameters derived from Atmospheric Infrared Sounder/Advanced Microwave Sounding Unit as a function of fractional cloud cover , 2006 .

[134]  Haruhisa Shimoda,et al.  Evaluation and improvement of SSM/I sea ice concentration algorithms for the Sea of Okhotsk , 1996 .

[135]  Didier Tanré,et al.  Evaluation of seven European aerosol optical depth retrieval algorithms for climate analysis , 2015 .

[136]  David R. Doelling,et al.  Toward Optimal Closure of the Earth's Top-of-Atmosphere Radiation Budget , 2009 .

[137]  Leif Toudal Pedersen,et al.  The impact of melt ponds on summertime microwave brightness temperatures and sea-ice concentrations , 2016 .

[138]  A. Kirkevåg,et al.  The Norwegian Earth System Model, NorESM1-M – Part 1: Description and basic evaluation of the physical climate , 2013 .

[139]  Elizabeth C. Kent,et al.  Global analyses of sea surface temperature, sea ice, and night marine air temperature since the late nineteenth century , 2003 .

[140]  Daniel R. Marsh,et al.  Climate change from 1850 to 2005 simulated in CESM1(WACCM) , 2013 .

[141]  David R. Doelling,et al.  Observed changes in top-of-the-atmosphere radiation and upper-ocean heating consistent within uncertainty , 2012 .

[142]  Eric Guilyardi,et al.  Towards improved and more routine Earth system model evaluation in CMIP , 2016 .

[143]  K. Denman,et al.  Carbon emission limits required to satisfy future representative concentration pathways of greenhouse gases , 2011 .