A framework for benchmarking land models

Land models, which have been developed by the modeling community in the past few decades to predict fu- ture states of ecosystems and climate, have to be critically evaluated for their performance skills of simulating ecosys- tem responses and feedback to climate change. Benchmark- ing is an emerging procedure to measure performance of models against a set of defined standards. This paper pro- poses a benchmarking framework for evaluation of land model performances and, meanwhile, highlights major chal- lenges at this infant stage of benchmark analysis. The frame- work includes (1) targeted aspects of model performance

[1]  J. S. Olson,et al.  Worldwide organic soil carbon and nitrogen data , 1986 .

[2]  Werner A. Kurz,et al.  Risk of natural disturbances makes future contribution of Canada's forests to the global carbon cycle highly uncertain , 2008, Proceedings of the National Academy of Sciences.

[3]  Benjamin Smith,et al.  CO2 fertilization in temperate FACE experiments not representative of boreal and tropical forests , 2008 .

[4]  Benjamin Z. Houlton,et al.  Nitrogen constraints on terrestrial carbon uptake: Implications for the global carbon‐climate feedback , 2009 .

[5]  Shenfeng Fei,et al.  Ecological forecasting and data assimilation in a data-rich era. , 2011, Ecological applications : a publication of the Ecological Society of America.

[6]  Eric F. Wood,et al.  Global estimates of evapotranspiration for climate studies using multi-sensor remote sensing data: Evaluation of three process-based approaches , 2011 .

[7]  Kuolin Hsu,et al.  Neural Error Regression Diagnosis (NERD): A Tool for Model Bias Identification and Prognostic Data Assimilation , 2006 .

[8]  J. Randerson,et al.  Continental-scale net radiation and evapotranspiration estimated using MODIS satellite observations , 2011 .

[9]  Charles D. Canham,et al.  Increased tree carbon storage in response to nitrogen deposition in the US , 2010 .

[10]  Hoshin Vijai Gupta,et al.  Systematic Bias in Land Surface Models , 2007 .

[11]  Pierre Friedlingstein,et al.  Terrestrial nitrogen feedbacks may accelerate future climate change , 2010 .

[12]  Damian J. Barrett,et al.  Estimating regional terrestrial carbon fluxes for the Australian continent using a multiple-constraint approach I. Using remotely sensed data and ecological observations of net primary production , 2003 .

[13]  Rachel M. Law,et al.  A global model of carbon, nitrogen and phosphorus cycles for the terrestrial biosphere , 2009 .

[14]  Shaun Quegan,et al.  Model–data synthesis in terrestrial carbon observation: methods, data requirements and data uncertainty specifications , 2005 .

[15]  D. Baldocchi,et al.  Global estimates of the land–atmosphere water flux based on monthly AVHRR and ISLSCP-II data, validated at 16 FLUXNET sites , 2008 .

[16]  M. R. Shaw,et al.  Water relations in grassland and desert ecosystems exposed to elevated atmospheric CO2 , 2004, Oecologia.

[17]  Zhiqiang Xiao,et al.  Reprocessing the MODIS Leaf Area Index products for land surface and climate modelling , 2011 .

[18]  Ray Leuning,et al.  Diagnosing errors in a land surface model (CABLE) in the time and frequency domains , 2011 .

[19]  K. Denman,et al.  Carbon emission limits required to satisfy future representative concentration pathways of greenhouse gases , 2011 .

[20]  Hans D. Mittelmann,et al.  A server for automated performance analysis of benchmarking data , 2006, Optim. Methods Softw..

[21]  Yong Wang,et al.  Using Model Trees for Classification , 1998, Machine Learning.

[22]  Markus Reichstein,et al.  Improving canopy processes in the Community Land Model version 4 (CLM4) using global flux fields empirically inferred from FLUXNET data , 2011 .

[23]  Maosheng Zhao,et al.  Development of a global evapotranspiration algorithm based on MODIS and global meteorology data , 2007 .

[24]  E. Matthews.  Global litter production, pools, and turnover times: Estimates from measurement data and regression models , 1997 .

[25]  R. Schnur,et al.  Climate-carbon cycle feedback analysis: Results from the C , 2006 .

[26]  D. Jolly,et al.  Mid‐Holocene and glacial‐maximum vegetation geography of the northern continents and Africa , 2000 .

[27]  Douglas C. Morton,et al.  Nitrogen deposition in tropical forests from savanna and deforestation fires , 2010 .

[28]  Christopher B. Field,et al.  Coordinated approaches to quantify long‐term ecosystem dynamics in response to global change , 2011 .

[29]  Peter A. Coppin,et al.  Parameter estimation in surface exchange models using nonlinear inversion: how many parameters can we estimate and which measurements are most useful? , 2001 .

[30]  Gordon B. Bonan,et al.  Quantifying carbon‐nitrogen feedbacks in the Community Land Model (CLM4) , 2010 .

[31]  Taotao Qian,et al.  Changes in Continental Freshwater Discharge from 1948 to 2004 , 2009 .

[32]  P. Heuberger,et al.  Calibration of process-oriented models , 1995 .

[33]  Ian G. Enting,et al.  A review of applications of model–data fusion to studies of terrestrial carbon fluxes at different scales , 2009 .

[34]  A. Robock,et al.  The International Soil Moisture Network: a data hosting facility for global in situ soil moisture measurements , 2011 .

[35]  K. Taylor.  Summarizing multiple aspects of model performance in a single diagram , 2001 .

[36]  M. Lefsky.  A global forest canopy height map from the Moderate Resolution Imaging Spectroradiometer and the Geoscience Laser Altimeter System , 2010 .

[37]  Peter E. Thornton,et al.  Systematic assessment of terrestrial biogeochemistry in coupled climate–carbon models , 2009 .

[38]  A. Hall,et al.  What Controls the Strength of Snow-Albedo Feedback? , 2007 .

[39]  J. Randerson,et al.  Technical Description of version 4.0 of the Community Land Model (CLM) , 2010 .

[40]  Damian Barrett,et al.  Estimating regional terrestrial carbon fluxes for the Australian continent using a multiple‐constraint approach , 2003 .

[41]  N. Oreskes.  2 The Role of Quantitative Models in Science , 2003 .

[42]  J. Canadell,et al.  Soil organic carbon pools in the northern circumpolar permafrost region , 2009 .

[43]  William G. Lee,et al.  Modulation of leaf economic traits and trait relationships by climate , 2005 .

[44]  John S. Kimball,et al.  Global Biomass Variation and Its Geodynamic Effects: 1982–98 , 2005 .

[45]  R. Ceulemans,et al.  Forest response to elevated CO2 is conserved across a broad range of productivity. , 2005, Proceedings of the National Academy of Sciences of the United States of America.

[46]  J. Berry,et al.  A biochemical model of photosynthetic CO2 assimilation in leaves of C3 species , 1980, Planta.

[47]  K. Rose,et al.  Model goodness-of-fit analysis using regression and related techniques , 1995 .

[48]  R. Dickinson,et al.  The Common Land Model , 2003 .

[49]  J. Chambers,et al.  Tree allometry and improved estimation of carbon stocks and balance in tropical forests , 2005, Oecologia.

[50]  Bo Li,et al.  Responses of ecosystem nitrogen cycle to nitrogen addition: a meta-analysis. , 2011, The New phytologist.

[51]  A. Pitman,et al.  Evaluating the Performance of Land Surface Models , 2008 .

[52]  Tyler A. Simon,et al.  Observation and analysis of the multicore performance impact on scientific applications , 2009 .

[53]  Yiqi Luo,et al.  Spatial patterns of ecosystem carbon residence time and NPP‐driven carbon uptake in the conterminous United States , 2008 .

[54]  Donald R. Zak,et al.  Ecological Lessons from Free-Air CO2 Enrichment (FACE) Experiments , 2011 .

[55]  R. Norby,et al.  Nitrogen uptake, distribution, turnover, and efficiency of use in a CO2-enriched sweetgum forest. , 2006, Ecology.

[56]  T. A. Boden,et al.  Worldwide Organic Soil Carbon and Nitrogen Data (1986) (NDP-018) , 1986 .

[57]  Vivek K. Arora,et al.  Estimating changes in global vegetation cover (1850–2100) for use in climate models , 2006 .

[58]  J. Peñuelas,et al.  Responses of terrestrial ecosystems to temperature and precipitation change: a meta‐analysis of experimental manipulation , 2011 .

[59]  J. Randerson,et al.  Continental-Scale Partitioning of Fire Emissions During the 1997 to 2001 El Niño/La Niña Period , 2003, Science.

[60]  Markus Reichstein,et al.  Reduction of forest soil respiration in response to nitrogen deposition , 2010 .

[61]  Jeffrey M. Warren,et al.  CO2 enhancement of forest productivity constrained by limited nitrogen availability , 2010, Proceedings of the National Academy of Sciences.

[62]  R. Dickinson,et al.  The Project for Intercomparison of Land Surface Parameterization Schemes (PILPS): Phases 2 and 3 , 1993 .

[63]  G. J. Collatz,et al.  ISLSCP II Global Primary Production Data Initiative Gridded NPP Data , 2011 .

[64]  Yiqi Luo,et al.  Divergence of reproductive phenology under climate warming , 2007, Proceedings of the National Academy of Sciences.

[65]  G. Bonan.  Forests and Climate Change: Forcings, Feedbacks, and the Climate Benefits of Forests , 2008, Science.

[66]  A. Pitman.  The evolution of, and revolution in, land surface schemes designed for climate models , 2003 .

[67]  Peter E. Thornton,et al.  Influence of carbon‐nitrogen cycle coupling on land model response to CO2 fertilization and climate variability , 2007 .

[68]  J. V. Soares,et al.  Distribution of aboveground live biomass in the Amazon basin , 2007 .

[69]  Stefan Hennemann.  Evaluating the performance of , 2012 .

[70]  S. Los,et al.  A comprehensive set of benchmark tests for a land surface model of simultaneous fluxes of water and carbon at both the global and seasonal scale , 2010 .

[71]  I. C. Prentice,et al.  Evaluation of the terrestrial carbon cycle, future plant geography and climate‐carbon cycle feedbacks using five Dynamic Global Vegetation Models (DGVMs) , 2008 .

[72]  S. Sitch,et al.  The role of fire disturbance for global vegetation dynamics: coupling fire into a Dynamic Global Vegetation Model , 2008 .

[73]  Tooraj Jamasb,et al.  International benchmarking and regulation: an application to European electricity distribution utilities , 2003 .

[74]  A. Arneth,et al.  Global patterns of land-atmosphere fluxes of carbon dioxide, latent heat, and sensible heat derived from eddy covariance, satellite, and meteorological observations , 2011 .

[75]  George H. Bryan,et al.  A Benchmark Simulation for Moist Nonhydrostatic Numerical Models , 2002 .

[76]  W. Knorr,et al.  Quantifying photosynthetic capacity and its relationship to leaf nitrogen content for global‐scale terrestrial biosphere models , 2009 .

[77]  Q. Zhuang,et al.  Equifinality in parameterization of process‐based biogeochemistry models: A significant uncertainty source to the estimation of regional carbon dynamics , 2008 .

[78]  Sandy P. Harrison,et al.  The influence of vegetation, fire spread and fire behaviour on biomass burning and trace gas emissions: results from a process-based model , 2010 .

[79]  David Dudgeon,et al.  A global experiment suggests climate warming will not accelerate litter decomposition in streams but might reduce carbon sequestration. , 2011, Ecology letters.

[80]  T. Carlson,et al.  On the relation between NDVI, fractional vegetation cover, and leaf area index , 1997 .

[81]  Yiqi Luo,et al.  Dynamic disequilibrium of the terrestrial carbon cycle under global change. , 2011, Trends in ecology & evolution.

[82]  L. White,et al.  Probabilistic inversion of a terrestrial ecosystem model: Analysis of uncertainty in parameter estimation and model prediction , 2006 .

[83]  Christopher B. Field,et al.  Nitrogen and Climate Change , 2003, Science.

[84]  Lingli Liu,et al.  A global perspective on belowground carbon dynamics under nitrogen enrichment. , 2010, Ecology letters.

[85]  Edward J. Rykiel,et al.  Testing ecological models: the meaning of validation , 1996 .

[86]  Hong Jiang,et al.  Integrating models with data in ecology and palaeoecology: advances towards a model-data fusion approach. , 2011, Ecology letters.

[87]  Yiqi Luo,et al.  Elevated CO2 stimulates net accumulations of carbon and nitrogen in land ecosystems: a meta-analysis. , 2006, Ecology.

[88]  W. Kurz,et al.  Mountain pine beetle and forest carbon feedback to climate change , 2008, Nature.

[89]  Gabriel Abramowitz,et al.  Towards a benchmark for land surface models , 2005 .

[90]  J. Randerson,et al.  Assessing variability and long-term trends in burned area by merging multiple satellite fire products , 2009 .

[91]  Jianyang Xia,et al.  Traceable components of terrestrial carbon storage capacity in biogeochemical models , 2013, Global change biology.

[92]  Yiqi Luo,et al.  FIRE EFFECTS ON NITROGEN POOLS AND DYNAMICS IN TERRESTRIAL ECOSYSTEMS: A META-ANALYSIS , 2001 .

[93]  Robert Jacob,et al.  Simulated and Observed Preindustrial to Modern Vegetation and Climate Changes , 2005 .

[94]  Yiqi Luo,et al.  inor stimulation of soil carbon storage by nitrogen addition : A meta-analysis , 2011 .

[95]  Nuno Carvalhais,et al.  Identification of vegetation and soil carbon pools out of equilibrium in a process model via eddy covariance and biometric constraints , 2010 .

[96]  H. Velthuizen,et al.  Harmonized World Soil Database (version 1.2) , 2008 .

[97]  Alan Hastings,et al.  Ecosystem carbon storage capacity as affected by disturbance regimes: A general theoretical model , 2012 .

[98]  W. J. Riley,et al.  Barriers to predicting changes in global terrestrial methane fluxes: analyses using CLM4Me, a methane biogeochemistry model integrated in CESM , 2011 .

[99]  Li Zhang,et al.  Parameter identifiability, constraint, and equifinality in data assimilation with ecosystem models. , 2009, Ecological applications : a publication of the Ecological Society of America.

[100]  P. Ciais,et al.  Changes in satellite‐derived vegetation growth trend in temperate and boreal Eurasia from 1982 to 2006 , 2011 .

[101]  R. Jeu,et al.  Multisensor historical climatology of satellite‐derived global land surface moisture , 2008 .

[102]  Peter B Reich,et al.  Predicting leaf physiology from simple plant and climate attributes: a global GLOPNET analysis. , 2007, Ecological applications : a publication of the Ecological Society of America.

[103]  S. Malyshev,et al.  The underpinnings of land‐use history: three centuries of global gridded land‐use transitions, wood‐harvest activity, and resulting secondary lands , 2006 .

[104]  Andrew D. Friend,et al.  Carbon and nitrogen cycle dynamics in the O‐CN land surface model: 1. Model description, site‐scale evaluation, and sensitivity to parameter estimates , 2010 .

[105]  P. Ciais,et al.  Benchmarking coupled climate‐carbon models against long‐term atmospheric CO2 measurements , 2010 .

[106]  C. Frankenberg,et al.  New global observations of the terrestrial carbon cycle from GOSAT: Patterns of plant fluorescence with gross primary productivity , 2011, Geophysical Research Letters.

[107]  Yiqi Luo,et al.  Carbon and nitrogen dynamics during forest stand development: a global synthesis. , 2011, The New phytologist.

[108]  J. Randerson,et al.  Interannual variability in global biomass burning emissions from 1997 to 2004 , 2006 .

[109]  Nuno Carvalhais,et al.  Comparing observations and process‐based simulations of biosphere‐atmosphere exchanges on multiple timescales , 2010 .

[110]  A. Baccini,et al.  Mapping forest canopy height globally with spaceborne lidar , 2011 .

[111]  Josep G. Canadell,et al.  Sustainability of terrestrial carbon sequestration: A case study in Duke Forest with inversion approach , 2003 .

[112]  Maosheng Zhao,et al.  Improvements to a MODIS global terrestrial evapotranspiration algorithm , 2011 .

[113]  S. Seneviratne,et al.  Recent decline in the global land evapotranspiration trend due to limited moisture supply , 2010, Nature.

[114]  Steven Platnick,et al.  Spatially complete global spectral surface albedos: value-added datasets derived from Terra MODIS land products , 2005, IEEE Transactions on Geoscience and Remote Sensing.

[115]  Yiqi Luo.  Terrestrial Carbon-Cycle Feedback to Climate Warming , 2007 .

[116]  I. C. Prentice,et al.  Evaluation of ecosystem dynamics, plant geography and terrestrial carbon cycling in the LPJ dynamic global vegetation model , 2003 .

[117]  Steven Platnick,et al.  MODIS-Derived Spatially Complete Surface Albedo Products: Spatial and Temporal Pixel Distribution and Zonal Averages , 2008 .

[118]  Stephen Sitch,et al.  Evaluation of terrestrial carbon cycle models with atmospheric CO2 measurements: Results from transient simulations considering increasing CO2, climate, and land‐use effects , 2002 .

[119]  Markus Reichstein,et al.  Consequences of More Extreme Precipitation Regimes for Terrestrial Ecosystems , 2008 .

[120]  Stephen Sitch,et al.  Evaluation of terrestrial carbon cycle models through simulations of the seasonal cycle of atmospheric CO2: First results of a model intercomparison study , 1998 .

[121]  S. Frolking,et al.  Canopy nitrogen, carbon assimilation, and albedo in temperate and boreal forests: Functional relations and potential climate feedbacks , 2008, Proceedings of the National Academy of Sciences.

[122]  Niels H. Batjes,et al.  Carbon and nitrogen stocks in the soils of Central and Eastern Europe , 2002 .

[123]  P. Friedlingstein,et al.  Modeling fire and the terrestrial carbon balance , 2011 .

[124]  Anjan K. Ghosh,et al.  Performance Analysis of Acousto-optic Digital Signal Processors using the Describing Function Approach , 1998 .

[125]  I. C. Prentice,et al.  Carbon balance of the terrestrial biosphere in the Twentieth Century: Analyses of CO2, climate and land use effects with four process‐based ecosystem models , 2001 .

[126]  Jesse E. Bell,et al.  Changes in soil water dynamics due to variation in precipitation and temperature: An ecohydrological analysis in a tallgrass prairie , 2010 .

[127]  Wilfred M. Post,et al.  Soil carbon pools and world life zones , 1982, Nature.

[128]  Gordon B. Bonan,et al.  Benefits of Forests Forests and Climate Change: Forcings, Feedbacks, and the Climate , 2014 .

[129]  Fabienne Maignan,et al.  Evaluation of a Global Vegetation Model using time series of satellite vegetation indices , 2011 .

[130]  S. Roxburgh,et al.  OptIC project: An intercomparison of optimization techniques for parameter estimation in terrestrial biogeochemical models , 2007 .

[131]  T. A. Black,et al.  A model‐data intercomparison of CO2 exchange across North America: Results from the North American Carbon Program site synthesis , 2010 .

[132]  Iain H. Woodhouse,et al.  Predicting backscatter-biomass and height-biomass trends using a macroecology model , 2006, IEEE Transactions on Geoscience and Remote Sensing.

[133]  Eleanor Blyth,et al.  Evaluating the JULES Land Surface Model Energy Fluxes Using FLUXNET Data , 2010 .

[134]  B. P. Bond-Lamberty,et al.  A Global Database of Soil Respiration Data, Version 1.0 , 2010 .

[135]  Yiqi Luo,et al.  Relative information contributions of model vs. data to short- and long-term forecasts of forest carbon dynamics. , 2011, Ecological applications : a publication of the Ecological Society of America.