ESMValTool ( v 1 . 0 ) – a community diagnostic and performance metrics tool for routine evaluation of Earth system models in

A community diagnostics and performance metrics tool for the evaluation of Earth system models (ESMs) has been developed that allows for routine comparison of single or multiple models, either against predecessor versions or against observations. The priority of the effort so far has been to target specific scientific themes focusing on selected essential climate variables (ECVs), a range of known systematic biases common to ESMs, such as coupled tropical climate variability, monsoons, Southern Ocean processes, continental dry biases, and soil hydrology–climate interactions, as well as atmospheric CO2 budgets, tropospheric and stratospheric ozone, and tropospheric aerosols. The tool is being developed in such a way that additional analyses can easily be added. A set of standard namelists for each scientific topic reproduces specific sets of diagnostics or performance metrics that have demonstrated their importance in ESM evaluation in the peer-reviewed literature. The Earth System Model Evaluation Tool (ESMValTool) is a community effort open to both users and developers encouraging open exchange of diagnostic source code and evaluation results from the Coupled Model Intercomparison Project (CMIP) ensemble. This will facilitate and improve ESM evaluation beyond the stateof-the-art and aims at supporting such activities within CMIP and at individual modelling centres. Ultimately, we envisage running the ESMValTool alongside the Earth System Grid Federation (ESGF) as part of a more routine evaluation of CMIP model simulations while utilizing observations available in standard formats (obs4MIPs) or provided by the user.

[1]  Duane E. Waliser,et al.  Satellite Observations for CMIP5: The Genesis of Obs4MIPs , 2014 .

[2]  Jialin Lin,et al.  The Double-ITCZ Problem in IPCC AR4 Coupled GCMs: Ocean–Atmosphere Feedback Analysis , 2007 .

[3]  T. Lavery,et al.  National Dry Deposition Network: Second annual progress report (1988) , 1990 .

[4]  P. Landschützer,et al.  Recent variability of the global ocean carbon sink , 2014 .

[5]  A. de Meij,et al.  Distributions and regional budgets of aerosols and their precursors simulated with the EMAC chemistry-climate model , 2012 .

[6]  P. Cox,et al.  Emergent constraints on climate‐carbon cycle feedbacks in the CMIP5 Earth system models , 2014 .

[7]  M. H. Savoie,et al.  A long-term and reproducible passive microwave sea ice concentration data record for climate studies and monitoring , 2013 .

[8]  Gerald G. Mace,et al.  Cloud properties and radiative forcing over the maritime storm tracks of the Southern Ocean and North Atlantic derived from A-Train , 2010 .

[9]  Hideyuki Shimizu,et al.  Major activities of acid deposition monitoring network in East Asia (EANET) and related studies , 2005 .

[10]  J. S. Olson,et al.  Major world ecosystem complexes ranked by carbon in live vegetation: a database , 1985 .

[11]  S. Seneviratne,et al.  Elusive drought: uncertainty in observed trends and short- and long-term CMIP5 projections , 2012 .

[12]  P. Xie,et al.  Global Precipitation: A 17-Year Monthly Analysis Based on Gauge Observations, Satellite Estimates, and Numerical Model Outputs , 1997 .

[13]  M. Holland,et al.  Arctic sea ice decline: Faster than forecast , 2007 .

[14]  James W. Hurrell,et al.  North Atlantic climate variability: The role of the North Atlantic Oscillation , 2009 .

[15]  Peter J. Webster,et al.  Recent change of the global monsoon precipitation (1979–2008) , 2012, Climate Dynamics.

[16]  Takao Iguchi.  Correlations between interannual variations of simulated global and regional CO2 fluxes from terrestrial ecosystems and El Niño Southern Oscillation , 2011 .

[17]  G. Balsamo,et al.  The WFDEI meteorological forcing data set: WATCH Forcing Data methodology applied to ERA‐Interim reanalysis data , 2014 .

[18]  A. Hall,et al.  Using the current seasonal cycle to constrain snow albedo feedback in future climate change , 2006 .

[19]  G. Bellon,et al.  The double ITCZ bias in CMIP5 models: interaction between SST, large-scale circulation and precipitation , 2015, Climate Dynamics.

[20]  Veronika Eyring,et al.  A community diagnostic tool for chemistry climate model validation , 2012 .

[21]  D. Legates,et al.  Mean seasonal and spatial variability in gauge‐corrected, global precipitation , 1990 .

[22]  John P. Krasting,et al.  Dominance of the Southern Ocean in Anthropogenic Carbon and Heat Uptake in CMIP5 Models , 2015 .

[23]  G. Louis Smith,et al.  Comparison of the Diurnal Cycle of Outgoing Longwave Radiation from a Climate Model with Results from ERBE , 2008 .

[24]  K. Taylor,et al.  Evaluating the present‐day simulation of clouds, precipitation, and radiation in climate models , 2008 .

[25]  Janet Sprintall,et al.  Southern Ocean mixed-layer depth from Argo float profiles , 2008 .

[26]  Peter J. Webster,et al.  Monsoon and Enso: Selectively Interactive Systems , 1992 .

[27]  Steven J. Ghan,et al.  Aerosol Properties and Processes: A Path from Field and Laboratory Measurements to Global Climate Models , 2007 .

[28]  Richard Neale,et al.  Application of MJO Simulation Diagnostics to Climate Models , 2009 .

[29]  John P. Dunne,et al.  Data‐based estimates of suboxia, denitrification, and N2O production in the ocean and their sensitivities to dissolved O2 , 2012 .

[30]  M. Webb,et al.  A quantitative performance assessment of cloud regimes in climate models , 2009 .

[31]  Antonio Navarra,et al.  Influence of ENSO and of the Indian Ocean Dipole on the Indian summer monsoon variability , 2013, Climate Dynamics.

[32]  Ingo Richter,et al.  What controls equatorial Atlantic winds in boreal spring? , 2014, Climate Dynamics.

[33]  Markus Stowasser,et al.  Response of the South Asian Summer Monsoon to Global Warming: Mean and Synoptic Systems , 2009 .

[34]  Stephen E. Schwartz,et al.  Observing and Modeling Earth’s Energy Flows , 2012, Surveys in Geophysics.

[35]  K. Findell,et al.  Simulation of Sahel drought in the 20th and 21st centuries. , 2005, Proceedings of the National Academy of Sciences of the United States of America.

[36]  Jean-Luc Redelsperger,et al.  The Present and Future of the West African Monsoon: A Process-Oriented Assessment of CMIP5 Simulations along the AMMA Transect , 2013 .

[37]  Patrick Jöckel,et al.  Development cycle 2 of the Modular Earth Submodel System (MESSy2) , 2010 .

[38]  Tim Li,et al.  Causes of Strengthening and Weakening of ENSO Amplitude under Global Warming in Four CMIP5 Models , 2015 .

[39]  S. Klein,et al.  Are climate model simulations of clouds improving? An evaluation using the ISCCP simulator , 2012 .

[40]  W. Connolley,et al.  An Antarctic assessment of IPCC AR4 coupled models , 2007 .

[41]  Ping Liu,et al.  Contrasting Madden–Julian Oscillation activity during various stages of EP and CP El Niños , 2015 .

[42]  Klaus Wyser,et al.  EC-Earth V2.2: description and validation of a new seamless earth system prediction model , 2012, Climate Dynamics.

[43]  D. Shindell,et al.  Anthropogenic and Natural Radiative Forcing , 2014 .

[44]  Veronika Eyring,et al.  Overview of the Coupled Model Intercomparison Project Phase 6 (CMIP6) experimental design and organization , 2015 .

[45]  W. Rossow,et al.  Advances in understanding clouds from ISCCP , 1999 .

[46]  A. Sterl,et al.  A look at the ocean in the EC-Earth climate model , 2012, Climate Dynamics.

[47]  U. Schneider,et al.  A description of the global land-surface precipitation data products of the Global Precipitation Climatology Centre with sample applications including centennial (trend) analysis from 1901–present , 2012 .

[48]  W. Paul Menzel,et al.  Cloud and aerosol properties, precipitable water, and profiles of temperature and water vapor from MODIS , 2003, IEEE Trans. Geosci. Remote. Sens..

[49]  B. Barkstrom,et al.  Clouds and the Earth's Radiant Energy System (CERES): An Earth Observing System Experiment , 1996 .

[50]  E. Guilyardi.  El Niño–mean state–seasonal cycle interactions in a multi-model ensemble , 2006 .

[51]  Pier Luigi Vidale,et al.  Evaluation of water and energy budgets in regional climate models applied over Europe , 2004 .

[52]  D. S. Pai,et al.  Impact of MJO on the intraseasonal variation of summer monsoon rainfall over India , 2011 .

[53]  N. Saji,et al.  Individual and Combined Influences of ENSO and the Indian Ocean Dipole on the Indian Summer Monsoon , 2004 .

[54]  S. Schubert,et al.  MERRA: NASA’s Modern-Era Retrospective Analysis for Research and Applications , 2011 .

[55]  Bruce R. Barkstrom,et al.  The Earth Radiation Budget Experiment (ERBE). , 1984 .

[56]  Reto Knutti,et al.  A Representative Democracy to Reduce Interdependency in a Multimodel Ensemble , 2015 .

[57]  Claudia Golz,et al.  Observed historical discharge data from major rivers for climate model validation , 2000 .

[58]  D. Hauglustaine,et al.  Data composites of airborne observations of tropospheric ozone and its precursors , 2000 .

[59]  A. Bondeau,et al.  Towards global empirical upscaling of FLUXNET eddy covariance observations: validation of a model tree ensemble approach using a biosphere model , 2009 .

[60]  Sang-Wook Yeh,et al.  The role of mean state on changes in El Niño’s flavor , 2011 .

[61]  Yi Ming,et al.  The general circulation model precipitation bias over the southwestern equatorial Indian Ocean and its implications for simulating the South Asian monsoon , 2013, Climate Dynamics.

[62]  Duane E. Waliser,et al.  Partitioning CloudSat ice water content for comparison with upper tropospheric ice in global atmospheric models , 2010 .

[63]  David R. Doelling,et al.  Observed changes in top-of-the-atmosphere radiation and upper-ocean heating consistent within uncertainty , 2012 .

[64]  Thomas J. Greenwald,et al.  The Earth's radiation budget and its relation to atmospheric hydrology: 1. Observations of the clear sky greenhouse effect , 1991 .

[65]  C. Deser,et al.  Communication of the role of natural variability in future North American climate , 2012 .

[66]  K. Taylor.  Summarizing multiple aspects of model performance in a single diagram , 2001 .

[67]  M. Kanamitsu,et al.  NCEP–DOE AMIP-II Reanalysis (R-2) , 2002 .

[68]  P. Jones,et al.  The Twentieth Century Reanalysis Project , 2009 .

[69]  Chris D. Jones,et al.  The Carbon Cycle Response to ENSO: A Coupled Climate–Carbon Cycle Model Study , 2001 .

[70]  Jochem Marotzke,et al.  Arctic sea‐ice evolution as modeled by Max Planck Institute for Meteorology's Earth system model , 2013 .

[71]  S. Xie,et al.  Tropical Biases in CMIP5 Multimodel Ensemble: The Excessive Equatorial Pacific Cold Tongue and Double ITCZ Problems* , 2014 .

[72]  Pawan K. Bhartia,et al.  A global climatology of tropospheric and stratospheric ozone derived from Aura OMI and MLS measurements , 2011 .

[73]  Alexander Loew,et al.  Combined evaluation of MPI‐ESM land surface water and energy fluxes , 2012 .

[74]  K.,et al.  Carbon–Concentration and Carbon–Climate Feedbacks in CMIP5 Earth System Models , 2012 .

[75]  Kiran Salunke,et al.  Simulation of boreal summer intraseasonal oscillations in the latest CMIP5 coupled GCMs , 2013 .

[76]  Clara Deser,et al.  Sea surface temperature variability: patterns and mechanisms. , 2010, Annual review of marine science.

[77]  Kevin E. Trenberth,et al.  An Observational Estimate of Inferred Ocean Energy Divergence , 2008 .

[78]  J. Lamarque,et al.  Technical Note: Ozonesonde climatology between 1995 and 2011: description, evaluation and applications , 2012 .

[79]  Dieter Gerten,et al.  Effects of Precipitation Uncertainty on Discharge Calculations for Main River Basins , 2009 .

[80]  J. Wallace,et al.  Annular Modes in the Extratropical Circulation. Part I: Month-to-Month Variability* , 2000 .

[81]  S. Bony,et al.  Climate change projections using the IPSL-CM5 Earth System Model: from CMIP3 to CMIP5 , 2013, Climate Dynamics.

[82]  J. Wallace,et al.  A Pacific Interdecadal Climate Oscillation with Impacts on Salmon Production , 1997 .

[83]  P. R. Julian,et al.  Detection of a 40–50 Day Oscillation in the Zonal Wind in the Tropical Pacific , 1971 .

[84]  M. Holland,et al.  Trends in Arctic sea ice extent from CMIP5, CMIP3 and observations , 2012 .

[85]  J. Lamarque,et al.  Pre-industrial to end 21st century projections of tropospheric ozone from the Atmospheric Chemistry and Climate Model Intercomparison Project (ACCMIP) , 2012 .

[86]  Bin Wang,et al.  The Asian summer monsoon: an intercomparison of CMIP5 vs. CMIP3 simulations of the late 20th century , 2013, Climate Dynamics.

[87]  M. Heimann,et al.  Interannual sea-air CO2 flux variability from an observation-driven ocean mixed-layer scheme , 2014 .

[88]  Raymond T. Pierrehumbert,et al.  Thermostats, Radiator Fins, and the Local Runaway Greenhouse , 1995 .

[89]  G. Madec.  NEMO ocean engine , 2008 .

[90]  Julia Slingo,et al.  The role of the basic state in the ENSO–monsoon relationship and implications for predictability , 2005 .

[91]  Kevin E. Trenberth,et al.  Atlantic hurricanes and natural variability in 2005 , 2006 .

[92]  Vasubandhu Misra,et al.  Daily atmospheric variability in the South American monsoon system , 2011 .

[93]  Taro Takahashi,et al.  Climatological distributions of pH, pCO2, total CO2, alkalinity, and CaCO3 saturation in the global surface ocean, and temporal changes at selected locations , 2014 .

[94]  Veronika Eyring,et al.  Chemistry-Climate Model Simulations of Twenty- First Century Stratospheric Climate and Circulation Changes , 2010 .

[95]  K. Cook,et al.  Coupled Model Simulations of the West African Monsoon System: Twentieth- and Twenty-First-Century Simulations , 2006 .

[96]  H. Eskes,et al.  Indicators of Antarctic ozone depletion , 2005 .

[97]  Mathew Barlow,et al.  Disruptions of El Niño–Southern Oscillation Teleconnections by the Madden–Julian Oscillation , 2014 .

[98]  W. Collins,et al.  The NCEP–NCAR 50-Year Reanalysis: Monthly Means CD-ROM and Documentation , 2001 .

[99]  V. Eyring,et al.  Quantitative evaluation of ozone and selected climate parameters in a set of EMAC simulations , 2014 .

[100]  Daehyun Kim,et al.  MJO and Convectively Coupled Equatorial Waves Simulated by CMIP5 Climate Models , 2013 .

[101]  Bin Wang,et al.  Choice of South Asian Summer Monsoon Indices , 1999 .

[102]  C. Deser,et al.  Projecting North American Climate over the Next 50 Years: Uncertainty due to Internal Variability* , 2014 .

[103]  S. Seneviratne,et al.  Systematic land climate and evapotranspiration biases in CMIP5 simulations , 2014, Geophysical research letters.

[104]  T. D. Mitchell,et al.  An improved method of constructing a database of monthly climate observations and associated high‐resolution grids , 2005 .

[105]  A. Lacis,et al.  Calculation of radiative fluxes from the surface to top of atmosphere based on ISCCP and other global data sets: Refinements of the radiative transfer model and the input data , 2004 .

[106]  T. Reichler,et al.  How Well Do Coupled Models Simulate Today's Climate? , 2008 .

[107]  B. Liebmann,et al.  Description of a complete (interpolated) outgoing longwave radiation dataset , 1996 .

[108]  Veronika Eyring,et al.  Evaluation of Climate Models. In: Climate Change 2013: The Physical Science Basis. Contribution of Working Group I to the Fifth Assessment Report of the Intergovernmental Panel on Climate Change , 2013 .

[109]  David R. Doelling,et al.  Toward Optimal Closure of the Earth's Top-of-Atmosphere Radiation Budget , 2009 .

[110]  W. Rossow,et al.  ISCCP Cloud Data Products , 1991 .

[111]  C. Thorncroft,et al.  African Monsoon Multidisciplinary Analysis: An International Research Project and Field Campaign , 2006 .

[112]  Brent N. Holben,et al.  An analysis of the collection 5 MODIS over-ocean aerosol optical depth product for its implication in aerosol assimilation , 2010 .

[113]  K. Trenberth,et al.  Simulation of Present-Day and Twenty-First-Century Energy Budgets of the Southern Oceans , 2010 .

[114]  V. Eyring,et al.  Quantitative performance metrics for stratospheric-resolving chemistry-climate models , 2008 .

[115]  Karl E. Taylor,et al.  An overview of CMIP5 and the experiment design , 2012 .

[116]  Volker Grewe,et al.  Earth System Chemistry integrated Modelling (ESCiMo) with the Modular Earth Submodel System (MESSy) version 2.51 , 2015 .

[117]  Frédéric Hourdin,et al.  Role of clouds and land‐atmosphere coupling in midlatitude continental summer warm biases and climate change amplification in CMIP5 simulations , 2014 .

[118]  Philip J. Klotzbach,et al.  The Madden-Julian Oscillation's Impacts on Worldwide Tropical Cyclone Activity , 2014 .

[119]  Robert Pincus,et al.  Reconciling Simulated and Observed Views of Clouds: MODIS, ISCCP, and the Limits of Instrument Simulators in Climate Models , 2011 .

[120]  C. Deser,et al.  Evaluating Modes of Variability in Climate Models , 2014 .

[121]  Reto Knutti,et al.  Addressing interdependency in a multimodel ensemble by interpolation of model properties , 2015 .

[122]  P. Cox,et al.  Evaluating the Land and Ocean Components of the Global Carbon Cycle in the CMIP5 Earth System Models , 2013 .

[123]  Ross Ihaka,et al.  Gentleman R: R: A language for data analysis and graphics , 1996 .

[124]  Eric D. Maloney,et al.  The Influence of the MJO on Upstream Precursors to African Easterly Waves , 2012 .

[125]  S. Bates,et al.  The CCSM4 Ocean Component , 2012 .

[126]  Charles Doutriaux,et al.  Performance metrics for climate models , 2008 .

[127]  Pierre Friedlingstein,et al.  Uncertainties in CMIP5 Climate Projections due to Carbon Cycle Feedbacks , 2014 .

[128]  F. Woodward,et al.  Terrestrial Gross Carbon Dioxide Uptake: Global Distribution and Covariation with Climate , 2010, Science.

[129]  B. Stevens,et al.  Climate and carbon cycle changes from 1850 to 2100 in MPI‐ESM simulations for the Coupled Model Intercomparison Project phase 5 , 2013 .

[130]  Robert A. Weller,et al.  Multidecade Global Flux Datasets from the Objectively Analyzed Air-sea Fluxes (OAFlux) Project: Latent and Sensible Heat Fluxes, Ocean Evaporation, and Related Surface Meteorological Variables , 2008 .

[131]  R. Schnur,et al.  Climate-carbon cycle feedback analysis: Results from the C , 2006 .

[132]  S. Schubert,et al.  Subseasonal variability associated with Asian summer monsoon simulated by 14 IPCC AR4 coupled GCMs , 2008 .

[133]  Axel Lauer,et al.  Simulating Clouds with Global Climate Models: A Comparison of CMIP5 Results with CMIP3 and Satellite Data , 2013 .

[134]  Massimo Bollasina,et al.  Indian Ocean SST, evaporation, and precipitation during the South Asian summer monsoon in IPCC-AR4 coupled simulations , 2009 .

[135]  Atul K. Jain,et al.  Global Carbon Budget 2018 , 2014, Earth System Science Data.

[136]  P. Cox,et al.  Sensitivity of tropical carbon to climate change constrained by carbon dioxide variability , 2013, Nature.

[137]  Veronika Eyring,et al.  Analysis of Present Day and Future OH and Methane Lifetime in the ACCMIP Simulations , 2012 .

[138]  Kevin Hamilton,et al.  The South Asian Summer Monsoon and Its Relationship with ENSO in the IPCC AR4 Simulations , 2007 .

[139]  Robert Sausen,et al.  The global impact of the transport sectors on atmospheric aerosol: simulations for year 2000 emissions , 2013 .

[140]  Gill Martin,et al.  The role of northern Arabian Sea surface temperature biases in CMIP5 model simulations and future projections of Indian summer monsoon rainfall , 2013, Climate Dynamics.

[141]  Frédéric Hourdin,et al.  Shifting the diurnal cycle of parameterized deep convection over land , 2009 .

[142]  Christopher W. O'Dell,et al.  Cloud Liquid Water Path from Satellite-Based Passive Microwave Observations: A New Climatology over the Global Oceans , 2008 .

[143]  T. Lebel,et al.  Mesoscale Convective System Rainfall in the Sahel , 2002 .

[144]  Nathaniel L. Bindoff,et al.  Detecting an external influence on recent changes in oceanic oxygen using an optimal fingerprinting method , 2012 .

[145]  S. Nicholson,et al.  An Analysis of Recent Rainfall Conditions in West Africa, Including the Rainy Seasons of the 1997 El Nino and the 1998 La Nina Years , 2000 .

[146]  S. Bony,et al.  The ‘too few, too bright’ tropical low‐cloud problem in CMIP5 models , 2012 .

[147]  P. Falkowski,et al.  Photosynthetic rates derived from satellite‐based chlorophyll concentration , 1997 .

[148]  Elizabeth C. Kent,et al.  Global analyses of sea surface temperature, sea ice, and night marine air temperature since the late nineteenth century , 2003 .

[149]  A. Timmermann,et al.  The Inverse Effect of Annual-Mean State and Annual-Cycle Changes on ENSO , 2010 .

[150]  J. Lamarque,et al.  Evaluation of preindustrial to present-day black carbon and its albedo forcing from Atmospheric Chemistry and Climate Model Intercomparison Project (ACCMIP) , 2012 .

[151]  U. Lohmann,et al.  Global indirect aerosol effects: a review , 2004 .

[152]  C. Tebaldi,et al.  Long-term Climate Change: Projections, Commitments and Irreversibility , 2013 .

[153]  J. Susskind,et al.  Global Precipitation at One-Degree Daily Resolution from Multisatellite Observations , 2001 .

[154]  A. Smirnov,et al.  AERONET-a federated instrument network and data archive for aerosol Characterization , 1998 .

[155]  Charles Doutriaux,et al.  A More Powerful Reality Test for Climate Models , 2016 .

[156]  M. Biasutti,et al.  Forced Sahel rainfall trends in the CMIP5 archive , 2013 .

[157]  Tsuyoshi Koshiro,et al.  Origins of the Solar Radiation Biases over the Southern Ocean in CFMIP2 Models , 2014 .