A comprehensive benchmarking system for evaluating global vegetation models

We present a benchmark system for global vegetation models. This system provides a quantitative evaluation of multiple simulated vegetation properties, including primary production; seasonal net ecosystem production; vegetation cover; composition and height; fire regime; and runoff. The benchmarks are derived from remotely sensed gridded datasets and site-based observations. The datasets allow comparisons of annual average conditions and seasonal and inter-annual variability, and they allow the impact of spatial and temporal biases in means and variability to be assessed separately. Specifically designed metrics quantify model performance for each process, and are compared to scores based on the temporal or spatial mean value of the observations and a "random" model produced by bootstrap resampling of the observations. The benchmark system is applied to three models: a simple light-use efficiency and water-balance model (the Simple Diagnostic Biosphere Model: SDBM), the Lund-Potsdam-Jena (LPJ) and Land Processes and eXchanges (LPX) dynamic global vegetation models (DGVMs). In general, the SDBM performs better than either of the DGVMs. It reproduces independent measurements of net primary production (NPP) but underestimates the amplitude of the observed CO 2 seasonal cycle. The two DGVMs show little difference for most benchmarks (including the inter-annual variability in the growth rate and seasonal cycle of atmospheric CO 2 ), but LPX represents burnt fraction demonstrably more accurately. Benchmarking also identified several weaknesses common to both DGVMs. The benchmarking system provides a quantitative approach for evaluating how adequately processes are represented in a model, identifying errors and biases, tracking improvements in performance through model development, and discriminating among models. Adoption of such a system would do much to improve confidence in terrestrial model predictions of climate change impacts and feedbacks.

[1]  R. Schnur,et al.  Climate-carbon cycle feedback analysis: Results from the C , 2006 .

[2]  R. DeFries,et al.  ISLSCP II Continuous Fields of Vegetation Cover, 1992-1993 , 2009 .

[3]  K. Taylor.  Summarizing multiple aspects of model performance in a single diagram , 2001 .

[4]  Jianhua Lu,et al.  A simulation and mechanism analysis of long-term variations at land surface over arid/semi-arid area in north China , 2006 .

[5]  Joanna M. Clark,et al.  Bioclimatic envelope model of climate change impacts on blanket peatland distribution in Great Britain , 2010 .

[6]  Wolfgang Knorr,et al.  Impact of drought stress and other factors on seasonal land biosphere CO2 exchange studied through an atmospheric tracer transport model , 1995 .

[7]  I. C. Prentice,et al.  Evaluation of ecosystem dynamics, plant geography and terrestrial carbon cycling in the LPJ dynamic global vegetation model , 2003 .

[8]  Sander Houweling,et al.  CO 2 flux history 1982–2001 inferred from atmospheric data using a global inversion of atmospheric transport , 2003 .

[9]  A. Ducharne,et al.  Development of a high resolution runoff routing model, calibration and application to assess runoff from the LMD GCM , 2003 .

[10]  J. Randerson,et al.  Terrestrial ecosystem production: A process model based on global satellite and surface data , 1993 .

[11]  A. Baccini,et al.  Mapping forest canopy height globally with spaceborne lidar , 2011 .

[12]  S. Roxburgh,et al.  OptIC project: An intercomparison of optimization techniques for parameter estimation in terrestrial biogeochemical models , 2007 .

[13]  M. Webb,et al.  Quantification of modelling uncertainties in a large ensemble of climate change simulations , 2004, Nature.

[14]  F. Woodward,et al.  Vegetation dynamics – simulating responses to climatic change , 2004, Biological reviews of the Cambridge Philosophical Society.

[15]  V. Haverd,et al.  CSIRO Marine and Atmospheric Research Component: Final Report for Phase 3 , 2008 .

[16]  Vivek K. Arora,et al.  Fire as an interactive component of dynamic vegetation models , 2005 .

[17]  S. Higgins,et al.  Impacts of climate change on the vegetation of Africa: an adaptive dynamic vegetation modelling approach , 2009 .

[18]  Steve Murray,et al.  Evaluation of global continental hydrology as simulated by the Land-surface Processes and eXchanges Dynamic Global Vegetation Model , 2010 .

[19]  S. Los,et al.  A comprehensive set of benchmark tests for a land surface model of simultaneous fluxes of water and carbon at both the global and seasonal scale , 2010 .

[20]  D. Baldocchi,et al.  Global estimates of the land–atmosphere water flux based on monthly AVHRR and ISLSCP-II data, validated at 16 FLUXNET sites , 2008 .

[21]  S. Seneviratne,et al.  Global Convergence in the Temperature Sensitivity of Respiration at Ecosystem Level , 2010, Science.

[22]  C. Parmesan.  Ecological and Evolutionary Responses to Recent Climate Change , 2006 .

[23]  J. Yeluripati,et al.  A Bayesian framework for model calibration, comparison and analysis: Application to four models for the biogeochemistry of a Norway spruce forest , 2011 .

[24]  R. Betts,et al.  Acceleration of global warming due to carbon-cycle feedbacks in a coupled climate model , 2000, Nature.

[25]  Peter E. Thornton,et al.  Systematic assessment of terrestrial biogeochemistry in coupled climate–carbon models , 2009 .

[26]  Aurel F. Moise,et al.  New climate model metrics based on object‐orientated pattern matching of rainfall , 2011 .

[27]  Corinne Le Quéré,et al.  Regional changes in carbon dioxide fluxes of land and oceans since 1980. , 2000, Science.

[28]  J. Randerson,et al.  Assessing variability and long-term trends in burned area by merging multiple satellite fire products , 2009 .

[29]  A. Bondeau,et al.  Towards global empirical upscaling of FLUXNET eddy covariance observations: validation of a model tree ensemble approach using a biosphere model , 2009 .

[30]  D. Gavin,et al.  A statistical approach to evaluating distance metrics and analog assignments for pollen records , 2003, Quaternary Research.

[31]  D. Roy,et al.  The distributions of a wide range of taxonomic groups are expanding polewards , 2006 .

[32]  Eleanor Blyth,et al.  Evaluating the JULES Land Surface Model Energy Fluxes Using FLUXNET Data , 2010 .

[33]  Maosheng Zhao,et al.  A Continuous Satellite-Derived Measure of Global Terrestrial Primary Production , 2004 .

[34]  W. Knorr,et al.  A climate-change risk analysis for world ecosystems , 2006, Proceedings of the National Academy of Sciences.

[35]  P. Ciais,et al.  Benchmarking coupled climate‐carbon models against long‐term atmospheric CO2 measurements , 2010 .

[36]  S. Pacala,et al.  A METHOD FOR SCALING VEGETATION DYNAMICS: THE ECOSYSTEM DEMOGRAPHY MODEL (ED) , 2001 .

[37]  J. Randerson,et al.  Interannual variability in global biomass burning emissions from 1997 to 2004 , 2006 .

[38]  Hisashi Nakamura,et al.  Application of Cluster Analysis to Climate Model Performance Metrics , 2011 .

[39]  N. Gobron,et al.  Monitoring the photosynthetic activity of vegetation from remote sensing data , 2006 .

[40]  H. Lambers,et al.  Carbon and nitrogen economy of 24 wild species differing in relative growth rate. , 1990, Plant physiology.

[41]  Philippe Ciais,et al.  A framework for benchmarking land models , 2012 .

[42]  J. Townshend,et al.  A new global 1‐km dataset of percentage tree cover derived from remote sensing , 2000 .

[43]  A. Bondeau,et al.  Comparing global models of terrestrial net primary productivity (NPP): overview and key results , 1999 .

[44]  M. Salam,et al.  Comparing Simulated and Measured Values Using Mean Squared Deviation and its Components , 2000 .

[45]  S. Sitch,et al.  The role of fire disturbance for global vegetation dynamics: coupling fire into a Dynamic Global Vegetation Model , 2008 .

[46]  Stephen Sitch,et al.  Dynamic global vegetation modelling: quantifying terrestrial ecosystem responses to large-scale environmental change , 2007 .

[47]  L. White,et al.  Probabilistic inversion of a terrestrial ecosystem model: Analysis of uncertainty in parameter estimation and model prediction , 2006 .

[48]  Thomas Kaminski,et al.  Sensitivity of the seasonal cycle of CO2 at remote monitoring stations with respect to seasonal surface exchange fluxes determined with the adjoint of an atmospheric transport model , 1996 .

[49]  Steven W. Running,et al.  Assessing interannual variation in MODIS-based estimates of gross primary production , 2006, IEEE Transactions on Geoscience and Remote Sensing.

[50]  J. Shukla,et al.  Climate model fidelity and projections of climate change , 2006 .

[51]  David Hinkley,et al.  Bootstrap Methods: Another Look at the Jackknife , 2008 .

[52]  S. Prince.  A model of regional primary production for use with coarse resolution satellite data , 1991 .

[53]  Charles Doutriaux,et al.  Performance metrics for climate models , 2008 .

[54]  F. Woodward,et al.  Terrestrial Gross Carbon Dioxide Uptake: Global Distribution and Covariation with Climate , 2010, Science.

[55]  Ralph F. Keeling,et al.  Recording Earth's Vital Signs , 2008, Science.

[56]  Markus Reichstein,et al.  CO2 balance of boreal, temperate, and tropical forests derived from a global database , 2007 .

[57]  Markus Reichstein,et al.  Improving canopy processes in the Community Land Model version 4 (CLM4) using global flux fields empirically inferred from FLUXNET data , 2011 .

[58]  Taotao Qian,et al.  Changes in Continental Freshwater Discharge from 1948 to 2004 , 2009 .

[59]  P. Friedlingstein,et al.  Modeling fire and the terrestrial carbon balance , 2011 .

[60]  J. Monteith.  SOLAR RADIATION AND PRODUCTIVITY IN TROPICAL ECOSYSTEMS , 1972 .

[61]  Maosheng Zhao,et al.  Improvements to a MODIS global terrestrial evapotranspiration algorithm , 2011 .

[62]  Sandy P. Harrison,et al.  The influence of vegetation, fire spread and fire behaviour on biomass burning and trace gas emissions: results from a process-based model , 2010 .

[63]  W. Lucht,et al.  Agricultural green and blue water consumption and its influence on the global water system , 2008 .

[64]  A. Hall,et al.  Using the current seasonal cycle to constrain snow albedo feedback in future climate change , 2006 .

[65]  Mrinal K. Sen,et al.  Error Reduction and Convergence in Climate Prediction , 2008 .

[66]  P. Dirmeyer,et al.  ISLSCP initiative II global data sets : Surface boundary conditions and atmospheric forcings for land-atmosphere studies , 2006 .

[67]  S. Prince,et al.  NPP Multi-Biome: NPP and Driver Data for Ecosystem Model-data Intercomparison, R2 , 2013 .

[68]  X. Zeng,et al.  Growing temperate shrubs over arid and semiarid regions in the Community Land Model–Dynamic Global Vegetation Model , 2008 .

[69]  C. Justice,et al.  Global distribution and seasonality of active fires as observed with the Terra and Aqua Moderate Resolution Imaging Spectroradiometer (MODIS) sensors , 2006 .

[70]  J. Randerson,et al.  Continental-Scale Partitioning of Fire Emissions During the 1997 to 2001 El Niño/La Niña Period , 2003, Science.

[71]  M. Allen,et al.  Constraints on climate change from a multi‐thousand member ensemble of simulations , 2005 .

[72]  Scott C. Doney,et al.  Variational data assimilation for atmospheric CO2 , 2006 .

[73]  I. C. Prentice,et al.  Evaluation of the terrestrial carbon cycle, future plant geography and climate‐carbon cycle feedbacks using five Dynamic Global Vegetation Models (DGVMs) , 2008 .

[74]  Kenneth L. Denman Canada.  Couplings between changes in the climate system and biogeochemistry , 2008 .

[75]  Fabienne Maignan,et al.  CO2 surface fluxes at grid point scale estimated from a global 21 year reanalysis of atmospheric measurements , 2010 .

[76]  Wolfgang Cramer,et al.  A simulation model for the transient effects of climate change on forest landscapes , 1993 .

[77]  Stephen Sitch,et al.  Evaluation of terrestrial carbon cycle models through simulations of the seasonal cycle of atmospheric CO2: First results of a model intercomparison study , 1998 .

[78]  J. Randerson,et al.  Global fire emissions and the contribution of deforestation, savanna, forest, agricultural, and peat fires (1997-2009) , 2010 .

[79]  W. Lucht,et al.  Terrestrial vegetation and water balance-hydrological evaluation of a dynamic global vegetation model , 2004 .

[80]  Scott C. Doney,et al.  Contribution of ocean, fossil fuel, land biosphere, and biomass burning carbon fluxes to seasonal and interannual variability in atmospheric CO2 , 2008 .

[81]  J. Townshend,et al.  Continuous fields of vegetation characteristics at the global scale at 1‐km resolution , 1999 .

[82]  A. Barnston.  Correspondence among the correlation, RMSE, and Heidke forecast verification measures; refinement of the Heidke score , 1992 .

[83]  J. Nash,et al.  River flow forecasting through conceptual models part I — A discussion of principles☆ , 1970 .

[84]  Robert Tibshirani,et al.  An Introduction to the Bootstrap , 1994 .

[85]  Two decades of ocean CO2 sink and variability , 2003 .

[86]  Geert Lenderink,et al.  Exploring metrics of extreme daily precipitation in a large ensemble of regional climate model simulations , 2010 .

[87]  Sung-Hyuk Cha.  Comprehensive Survey on Distance/Similarity Measures between Probability Density Functions , 2007 .

[88]  T. Reichler,et al.  How Well Do Coupled Models Simulate Today's Climate? , 2008 .

[89]  Kevin E. Trenberth,et al.  Estimates of Freshwater Discharge from Continents: Latitudinal and Seasonal Variations , 2002 .

[90]  Mark Rounsevell,et al.  Climate Change 2007: Impacts, Adaptation and Vulnerability. Contribution of Working Group II to the Fourth Assessment Report of the Intergovernmental Panel on Climate Change , 2007 .

[91]  J. Randerson,et al.  Systematic assessment of terrestrial biogeochemistry in coupled climate–carbon models , 2009 .

[92]  David Riaño,et al.  Burned area forecasting using past burned area records and Southern Oscillation Index for tropical Africa (1981–1999) , 2007 .

[93]  Yiqi Luo,et al.  Relative information contributions of model vs. data to short- and long-term forecasts of forest carbon dynamics. , 2011, Ecological applications : a publication of the Ecological Society of America.

[94]  Joshua B. Fisher,et al.  ET come home: potential evapotranspiration in geographical ecology , 2011 .

[95]  S. Seneviratne,et al.  Recent decline in the global land evapotranspiration trend due to limited moisture supply , 2010, Nature.

[96]  A. Belward,et al.  Characterizing interannual variations in global fire calendar using data from Earth observing satellites , 2005 .