Land surface Verification Toolkit (LVT) – a generalized framework for land surface model evaluation

Abstract. Model evaluation and verification are key in improving the usage and applicability of simulation models for real-world applications. In this article, the development and capabilities of a formal system for land surface model evaluation called the Land surface Verification Toolkit (LVT) is described. LVT is designed to provide an integrated environment for systematic land model evaluation and facilitates a range of verification approaches and analysis capabilities. LVT operates across multiple temporal and spatial scales and employs a large suite of in-situ, remotely sensed and other model and reanalysis datasets in their native formats. In addition to the traditional accuracy-based measures, LVT also includes uncertainty and ensemble diagnostics, information theory measures, spatial similarity metrics and scale decomposition techniques that provide novel ways for performing diagnostic model evaluations. Though LVT was originally designed to support the land surface modeling and data assimilation framework known as the Land Information System (LIS), it supports hydrological data products from non-LIS environments as well. In addition, the analysis of diagnostics from various computational subsystems of LIS including data assimilation, optimization and uncertainty estimation are supported within LVT. Together, LIS and LVT provide a robust end-to-end environment for enabling the concepts of model data fusion for hydrological applications. The evolving capabilities of LVT framework are expected to facilitate rapid model evaluation efforts and aid the definition and refinement of formal evaluation procedures for the land surface modeling community.

[1]  Yudong Tian,et al.  Estimating evapotranspiration with land data assimilation systems , 2011 .

[2]  M. Seyfried,et al.  Scale and the Nature of Spatial Variability: Field Examples Having Implications for Hydrologic Modeling , 1995 .

[3]  Wade T. Crow,et al.  Performance Metrics for Soil Moisture Retrievals and Application Requirements , 2009 .

[4]  Kenneth W. Harrison,et al.  A comparison of methods for a priori bias correction in soil moisture data assimilation , 2012 .

[5]  Yuqiong Liu,et al.  A wavelet-based approach to assessing timing errors in hydrologic predictions , 2011 .

[6]  A. H. Murphy,et al.  Diagnostic verification of probability forecasts , 1992 .

[7]  Hoshin Vijai Gupta,et al.  Decomposition of the mean squared error and NSE performance criteria: Implications for improving hydrological modelling , 2009 .

[8]  Eric F. Wood,et al.  Scale Problems in Hydrology , 1986 .

[9]  Sujay V. Kumar,et al.  An integrated high-resolution hydrometeorological modeling testbed using LIS and WRF , 2008, Environ. Model. Softw..

[10]  Lifeng Luo,et al.  Snow process modeling in the North American Land Data Assimilation System (NLDAS): 1. Evaluation of model‐simulated snow cover extent , 2003 .

[11]  Naota Hanasaki,et al.  GSWP-2 Multimodel Analysis and Implications for Our Perception of the Land Surface , 2006 .

[12]  Youlong Xia,et al.  Drought Indices Based on the Climate Forecast System Reanalysis and Ensemble NLDAS , 2011 .

[13]  Paul R. Houser,et al.  An Integrated Hydrologic Modeling and Data Assimilation Framework , 2008 .

[14]  Jan Polcher,et al.  Acceleration of Land Surface Model Development over a Decade of Glass , 2011 .

[15]  A. Bondeau,et al.  Towards global empirical upscaling of FLUXNET eddy covariance observations: validation of a model tree ensemble approach using a biosphere model , 2009 .

[16]  Keith Beven,et al.  Effects of spatial variability and scale with implications to hydrologic modeling , 1988 .

[17]  S. Seneviratne,et al.  Global intercomparison of 12 land surface heat flux estimates , 2011 .

[18]  Christa D. Peters-Lidard,et al.  A Modeling and Observational Framework for Diagnosing Local Land–Atmosphere Coupling on Diurnal Time Scales , 2009 .

[19]  Benny Selle,et al.  Effective landscape modelling using CART and complexity measures , 2004 .

[20]  N. Kiang,et al.  Land Surface Model Development for the GISS GCM: Effects of Improved Canopy Physiology on Simulated Climate , 2005 .

[21]  S. Los,et al.  A comprehensive set of benchmark tests for a land surface model of simultaneous fluxes of water and carbon at both the global and seasonal scale , 2010 .

[22]  Randal D. Koster,et al.  On the Nature of Soil Moisture in Land Surface Models , 2009 .

[23]  G. Niu,et al.  Model performance, model robustness, and model fitness scores: A new method for identifying good land‐surface models , 2008 .

[24]  Boleslaw K. Szymanski,et al.  High-Performance Object-Oriented Scientific Programming in Fortran 90 , 1997, PPSC.

[25]  Sujay V. Kumar,et al.  High-performance Earth system modeling with NASA/GSFC’s Land Information System , 2007, Innovations in Systems and Software Engineering.

[26]  M. Williams,et al.  Improving land surface models with FLUXNET data , 2009 .

[27]  K. Mitchell,et al.  A parameterization of snowpack and frozen ground intended for NCEP weather and climate models , 1999 .

[28]  R. Dickinson,et al.  The Common Land Model , 2003 .

[29]  Yuqiong Liu,et al.  Reconciling theory with observations: elements of a diagnostic approach to model evaluation , 2008 .

[30]  Holger Lange,et al.  Are Ecosystems Dynamical Systems , 1999 .

[31]  Wade T. Crow,et al.  An adaptive ensemble Kalman filter for soil moisture data assimilation , 2007 .

[32]  Maosheng Zhao,et al.  Development of a global evapotranspiration algorithm based on MODIS and global meteorology data , 2007 .

[33]  A. Pitman,et al.  Evaluating the Performance of Land Surface Models , 2008 .

[34]  Rodger B. Grayson,et al.  Quantitative comparison of spatial fields for hydrological model assessment––some promising approaches , 2005 .

[35]  S. Seneviratne,et al.  Basin scale estimates of evapotranspiration using GRACE and other observations , 2004 .

[36]  Jouni Pulliainen,et al.  Mapping of snow water equivalent and snow depth in boreal and sub-arctic zones by assimilating space-borne microwave radiometer data and ground-based observations , 2006 .

[37]  J. Kurths,et al.  A Comparative Classification of Complexity Measures , 1994 .

[38]  D. Stephenson,et al.  A new intensity‐scale approach for the verification of spatial precipitation forecasts , 2004 .

[39]  Peter E. Thornton,et al.  Systematic assessment of terrestrial biogeochemistry in coupled climate–carbon models , 2009 .

[40]  Kelly Elder,et al.  Scaling properties and spatial organization of snow depth fields in sub‐alpine forest and alpine tundra , 2009 .

[41]  Murugesu Sivapalan,et al.  Scale issues in hydrological modelling: A review , 1995 .

[42]  J. D. Tarpley,et al.  The multi‐institution North American Land Data Assimilation System (NLDAS): Utilizing multiple GCIP products and partners in a continental distributed hydrological modeling system , 2004 .

[43]  J. D. Tarpley,et al.  Implementation of Noah land surface model advances in the National Centers for Environmental Prediction operational mesoscale Eta model , 2003 .

[44]  Jeffrey P. Walker,et al.  THE GLOBAL LAND DATA ASSIMILATION SYSTEM , 2004 .

[45]  Kenneth W. Harrison,et al.  Quantifying the change in soil moisture modeling uncertainty from remote sensing observations using Bayesian inference techniques , 2012 .

[46]  T. Meyers,et al.  Sensitivity of Land Surface Simulations to Model Physics, Land Characteristics, and Forcings, at Four CEOP Sites , 2007 .

[47]  Stephen J. Lord,et al.  The New Global Operational Analysis System at the National Meteorological Center , 1991 .

[48]  Günter Blöschl,et al.  Scaling issues in snow hydrology , 1999 .

[49]  G. Gayno,et al.  Implementation of Noah land-surface model advances in the NCEP operational mesoscale Eta model , 2003 .

[50]  A.H. Haddad,et al.  Applied optimal estimation , 1976, Proceedings of the IEEE.

[51]  J. D. Tarpley,et al.  Evaluation of the North American Land Data Assimilation System over the southern Great Plains during the warm season , 2003 .

[52]  Eleanor Blyth,et al.  Evaluating the JULES Land Surface Model Energy Fluxes Using FLUXNET Data , 2010 .

[53]  M. R. R A U Pa C H,et al.  Model – data synthesis in terrestrial carbon observation : methods , data requirements and data uncertainty specifications , 2005 .

[54]  P. Xie,et al.  Global Precipitation: A 17-Year Monthly Analysis Based on Gauge Observations, Satellite Estimates, and Numerical Model Outputs , 1997 .

[55]  T. Meixner.  Spatial Patterns in Catchment Hydrology: Observations and Modelling , 2002 .

[56]  J. Randerson,et al.  Systematic assessment of terrestrial biogeochemistry in coupled climate–carbon models , 2009 .

[57]  Yakov A. Pachepsky,et al.  Information and complexity measures applied to observed and simulated soil moisture time series , 2011 .

[58]  Thomas J. Jackson,et al.  Soil moisture retrieval from AMSR-E , 2003, IEEE Trans. Geosci. Remote. Sens..

[59]  R. Koster,et al.  Comparison and assimilation of global soil moisture retrievals from the Advanced Microwave Scanning Radiometer for the Earth Observing System (AMSR‐E) and the Scanning Multichannel Microwave Radiometer (SMMR) , 2007 .

[60]  Lifeng Luo,et al.  Snow process modeling in the north american Land Data Assimilation System (NLDAS): 2. Evaluation of model simulated snow water equivalent : GEWEX Continental-Scale International Project, Part 3 (GCIP3) , 2003 .

[61]  Lifeng Luo,et al.  Streamflow and water balance intercomparisons of four land surface models in the North American Land Data Assimilation System Project , 2004 .

[62]  K. Beven,et al.  Similarity and scale in catchment storm response , 1990 .

[63]  Wade T. Crow,et al.  Comparison of adaptive filtering techniques for land surface data assimilation , 2008 .

[64]  D. Lawrence,et al.  Parameterization improvements and functional and structural advances in Version 4 of the Community Land Model , 2011 .

[65]  Boleslaw K. Szymanski,et al.  How to Express C++ Concepts in Fortran90 , 1997, Sci. Program..

[66]  R. Heim.  A Review of Twentieth-Century Drought Indices Used in the United States , 2002 .

[67]  Ann Henderson-Sellers,et al.  Recent progress and results from the project for the intercomparison of landsurface parameterization schemes , 1998 .

[68]  Eli Skop,et al.  Scale Issues in Hydrological Modeling , 1996 .

[69]  Ann Henderson-Sellers,et al.  The Project for Intercomparison of Land Surface Parameterization Schemes (PILPS): Phases 2 and 3 , 1995 .

[70]  M. Ek,et al.  Continental‐scale water and energy flux analysis and validation for North American Land Data Assimilation System project phase 2 (NLDAS‐2): 2. Validation of model‐simulated streamflow , 2012 .

[71]  Oregon Entomological,et al.  Bulletin of the , 2013 .

[72]  K. Mo,et al.  Continental-scale water and energy flux analysis and validation for the North American Land Data Assimilation System project phase 2 (NLDAS-2): 1. Intercomparison and application of model products , 2012 .

[73]  M. S. Moran,et al.  Using remotely-sensed estimates of soil moisture to infer soil texture and hydraulic properties across a semi-arid watershed , 2007 .

[74]  Aaron A. Berg,et al.  Realistic Initialization of Land Surface States: Impacts on Subseasonal Forecast Skill , 2004 .

[75]  A. Barrett,et al.  National Operational Hydrologic Remote Sensing Center SNOw Data Assimilation System (SNODAS) Products at NSIDC , 2003 .

[76]  Wade T. Crow,et al.  The Contributions of Precipitation and Soil Moisture Observations to the Skill of Soil Moisture Estimates in a Land Data Assimilation System , 2011 .

[77]  Yakov A. Pachepsky,et al.  Information content and complexity of simulated soil water fluxes , 2006 .

[78]  Sujay V. Kumar,et al.  Land information system: An interoperable framework for high resolution land surface modeling , 2006, Environ. Model. Softw..

[79]  W. J. Shuttleworth,et al.  An Agenda for Land-Surface Hydrology Research and a Call for the Second International Hydrological Decade , 2013 .

[80]  Edward J. Kim,et al.  A blended global snow product using visible, passive microwave and scatterometer satellite data , 2011 .

[81]  T. Erickson,et al.  Persistence of topographic controls on the spatial distribution of snow in rugged mountain terrain, Colorado, United States , 2005 .

[82]  Yuqiong Liu,et al.  The Ensemble Verification System (EVS): A software tool for verifying ensemble forecasts of hydrometeorological and hydrologic variables at discrete locations , 2010, Environ. Model. Softw..

[83]  R. Jeu,et al.  Multisensor historical climatology of satellite‐derived global land surface moisture , 2008 .

[84]  C. Rosenzweig,et al.  Increased crop damage in the US from excess precipitation under climate change , 2002 .

[85]  Wade T. Crow,et al.  A land surface data assimilation framework using the land information system : Description and applications , 2008 .

[86]  Cecelia DeLuca,et al.  The architecture of the Earth System Modeling Framework , 2003, Computing in Science & Engineering.

[87]  W. Rossow,et al.  ISCCP Cloud Data Products , 1991 .

[88]  Günter Blöschl,et al.  Spatial Patterns of Catchment Hydrology: Observations and Modelling , 2000 .

[89]  M. Sivapalan,et al.  Scale problems in hydrology: Contributions of the robertson workshop , 1995 .

[90]  M. S. Moran,et al.  Role of precipitation uncertainty in the estimation of hydrologic soil properties using remotely sensed soil moisture in a semiarid environment , 2008 .

[91]  Murugesu Sivapalan,et al.  Scale issues in hydrological modelling , 1995 .

[92]  J. Shukla,et al.  Impact of Initial Soil Wetness on Seasonal Atmospheric Prediction , 1999 .

[93]  Christa D. Peters-Lidard,et al.  Diagnosing the Sensitivity of Local Land–Atmosphere Coupling via the Soil Moisture–Boundary Layer Interaction , 2011 .

[94]  Y. Xue,et al.  Modeling of land surface evaporation by four schemes and comparison with FIFE observations , 1996 .

[95]  J. Dudhia,et al.  Noah land surface model modifications to improve snowpack prediction in the Colorado Rocky Mountains , 2010 .

[96]  Kevin W. Manning,et al.  The community Noah land surface model with multiparameterization options (Noah-MP): 1. Model description and evaluation with local-scale measurements , 2011 .