Introductory overview of identifiability analysis: A guide to evaluating whether you have the right type of data for your modeling purpose

Identifiability is a fundamental concept in parameter estimation, and therefore key to the large majority of environmental modeling applications. Parameter identifiability analysis assesses whether it is theoretically possible to estimate unique parameter values from data, given the quantities measured, conditions present in the forcing data, model structure (and objective function), and properties of errors in the model and observations. In other words, it tackles the problem of whether the right type of data is available to estimate the desired parameter values. Identifiability analysis is therefore an essential technique that should be adopted more routinely in practice, alongside complementary methods such as uncertainty analysis and evaluation of model performance. This article provides an introductory overview to the topic. We recommend that any modeling study should document whether a model is non-identifiable, the source of potential non-identifiability, and how this affects intended project outcomes.

[1]  Martin F. Lambert,et al.  Calibration and validation of neural networks to ensure physically plausible hydrological modeling , 2005 .

[2]  Charbel Farhat,et al.  A nonparametric probabilistic approach for quantifying uncertainties in low‐dimensional and high‐dimensional nonlinear models , 2017 .

[3]  J. P. Norton,et al.  An Introduction to Identification , 1986 .

[4]  Bruno Sudret,et al.  Global sensitivity analysis using polynomial chaos expansions , 2008, Reliab. Eng. Syst. Saf..

[5]  Emmanuel D. Blanchard,et al.  Polynomial chaos-based parameter estimation methods applied to a vehicle system , 2010 .

[6]  Xiaohua Xia,et al.  On Identifiability of Nonlinear ODE Models and Applications in Viral Dynamics , 2011, SIAM Rev..

[7]  A. Tikhonov,et al.  Numerical Methods for the Solution of Ill-Posed Problems , 1995 .

[8]  B. Croke,et al.  A review of foundational methods for checking the structural identifiability of models: Results for rainfall-runoff , 2015 .

[9]  Saman Razavi,et al.  VARS-TOOL: A toolbox for comprehensive, efficient, and robust sensitivity and uncertainty analysis , 2019, Environ. Model. Softw..

[10]  K. R. Godfrey,et al.  Chapter 1 – IDENTIFIABILITY OF MODEL PARAMETERS , 1987 .

[11]  M.P. Saccomani,et al.  DAISY: An efficient tool to test global identifiability. Some case studies , 2008, 2008 16th Mediterranean Conference on Control and Automation.

[12]  Demetris Koutsoyiannis,et al.  One decade of multi-objective calibration approaches in hydrological modelling: a review , 2010 .

[13]  Keith Beven,et al.  A manifesto for the equifinality thesis , 2006 .

[14]  S. P. Neuman.  Calibration of distributed parameter groundwater flow models viewed as a multiple‐objective decision process under uncertainty , 1973 .

[15]  Anthony J. Jakeman,et al.  Ten iterative steps in development and evaluation of environmental models , 2006, Environ. Model. Softw..

[16]  Peter A. Vanrolleghem,et al.  Uncertainty in the environmental modelling process - A framework and guidance , 2007, Environ. Model. Softw..

[17]  Joseph H. A. Guillaume,et al.  Characterising performance of environmental models , 2013, Environ. Model. Softw..

[18]  George Kuczera,et al.  Critical evaluation of parameter consistency and predictive uncertainty in hydrological modeling: A case study using Bayesian total error analysis , 2009 .

[19]  L. Shawn Matott,et al.  Evaluating uncertainty in integrated environmental models: A review of concepts and tools , 2009 .

[20]  C. Cobelli,et al.  Parameter and structural identifiability concepts and ambiguities: a critical review and analysis. , 1980, The American journal of physiology.

[21]  A. O'Hagan,et al.  Probabilistic sensitivity analysis of complex models: a Bayesian approach , 2004 .

[22]  J. P. Norton,et al.  Normal-mode identifiability analysis of linear compartmental systems in linear stages , 1980 .

[23]  Thierry Bastogne,et al.  Limits of variance-based sensitivity analysis for non-identifiability testing in high dimensional dynamic models , 2012, Autom..

[24]  Soroosh Sorooshian,et al.  Toward improved calibration of hydrologic models: Multiple and noncommensurable measures of information , 1998 .

[25]  Dmitri Kavetski,et al.  Pursuing the method of multiple working hypotheses for hydrological modeling , 2011 .

[26]  Karel J. Keesman,et al.  Direct least-squares estimation and prediction of rational systems: Application to food storage , 2009 .

[27]  M. B. Beck,et al.  Water quality modeling: A review of the analysis of uncertainty , 1987 .

[28]  Dmitri Kavetski,et al.  Practical Use of Computationally Frugal Model Analysis Methods , 2016, Ground water.

[29]  J. P. Norton,et al.  An investigation of the sources of nonuniqueness in deterministic identifiability , 1982 .

[30]  Emanuele Borgonovo,et al.  Making the most out of a hydrological model data set: Sensitivity analyses to open the model black‐box , 2017 .

[31]  Johan Karlsson,et al.  An Efficient Method for Structural Identifiability Analysis of Large Dynamic Systems , 2012 .

[32]  Saman Razavi,et al.  Insights into sensitivity analysis of Earth and environmental systems models: On the impact of parameter perturbation scale , 2017, Environ. Model. Softw..

[33]  Neil McIntyre,et al.  Towards reduced uncertainty in conceptual rainfall‐runoff modelling: dynamic identifiability analysis , 2003 .

[34]  Eva Balsa-Canto,et al.  Bioinformatics Applications Note Systems Biology Genssi: a Software Toolbox for Structural Identifiability Analysis of Biological Models , 2022 .

[35]  Günter Blöschl,et al.  Advances in the use of observed spatial patterns of catchment hydrological response , 2002 .

[36]  Paola Annoni,et al.  Sixth International Conference on Sensitivity Analysis of Model Output How to avoid a perfunctory sensitivity analysis , 2010 .

[37]  Joseph H. A. Guillaume,et al.  Toward best practice framing of uncertainty in scientific publications: A review of Water Resources Research abstracts , 2017 .

[38]  P. Brunner,et al.  Beyond Classical Observations in Hydrogeology: The Advantages of Including Exchange Flux, Temperature, Tracer Concentration, Residence Time, and Soil Moisture Observations in Groundwater Model Calibration , 2019, Reviews of Geophysics.

[39]  J. Vrugt,et al.  A formal likelihood function for parameter and predictive inference of hydrologic models with correlated, heteroscedastic, and non‐Gaussian errors , 2010 .

[40]  Kevin Carlberg,et al.  Adaptive h‐refinement for reduced‐order models , 2014, ArXiv.

[41]  M. B. Beck,et al.  Assessing local structural identifiability for environmental models , 2017, Environ. Model. Softw..

[42]  Ursula Klingmüller,et al.  Structural and practical identifiability analysis of partially observed dynamical models by exploiting the profile likelihood , 2009, Bioinform..

[43]  Ming Ye,et al.  Towards a comprehensive assessment of model structural adequacy , 2012 .

[44]  Jennifer Badham,et al.  Effective modeling for Integrated Water Resource Management: A guide to contextual practices by phases and steps and future opportunities , 2019, Environ. Model. Softw..

[45]  J. Doherty,et al.  Uncertainty assessment and implications for data acquisition in support of integrated hydrologic models , 2012 .

[46]  T. Koopmans,et al.  The Identification of Structural Characteristics , 1950 .

[47]  Mary C. Hill,et al.  Distributed Evaluation of Local Sensitivity Analysis (DELSA), with application to hydrologic models , 2014 .

[48]  S. Marsili-Libelli,et al.  Confidence regions of estimated parameters for ecological systems , 2003 .

[49]  Soroosh Sorooshian,et al.  A framework for development and application of hydrological models , 2001, Hydrology and Earth System Sciences.

[50]  Stefano Marsili-Libelli,et al.  Parameter estimation of ecological models , 1992 .

[51]  C. Tiedeman,et al.  Effective Groundwater Model Calibration: With Analysis of Data, Sensitivities, Predictions, and Uncertainty , 2007 .

[52]  Byron K Williams,et al.  Passive and active adaptive management: approaches and an example. , 2011, Journal of environmental management.

[53]  Holger R. Maier,et al.  Review of Input Variable Selection Methods for Artificial Neural Networks , 2011 .

[54]  Jasper A. Vrugt,et al.  Markov chain Monte Carlo simulation using the DREAM software package: Theory, concepts, and MATLAB implementation , 2016, Environ. Model. Softw..

[55]  John D. Jakeman,et al.  Optimal Experimental Design Using A Consistent Bayesian Approach , 2017, 1705.09395.

[56]  Saman Razavi,et al.  Revisiting the Basis of Sensitivity Analysis for Dynamical Earth System Models , 2018, Water Resources Research.

[57]  T. Rothenberg.  Identification in Parametric Models , 1971 .

[58]  Bryan A. Tolson,et al.  Review of surrogate modeling in water resources , 2012 .

[59]  Saman Razavi,et al.  What do we mean by sensitivity analysis? The need for comprehensive characterization of “global” sensitivity in Earth and Environmental systems models , 2015 .

[60]  Joel Massmann,et al.  Hydrogeological Decision Analysis: 4. The Concept of Data Worth and Its Use in the Development of Site Investigation Strategies , 1992 .

[61]  Eric Walter,et al.  On the identifiability and distinguishability of nonlinear parametric models , 1996 .

[62]  Hubert H. G. Savenije,et al.  Using expert knowledge to increase realism in environmental system models can dramatically reduce the need for calibration , 2013 .

[63]  Saman Razavi,et al.  A multi-method Generalized Global Sensitivity Matrix approach to accounting for the dynamical nature of earth and environmental systems models , 2019, Environ. Model. Softw..

[64]  Andrea Saltelli,et al.  An effective screening design for sensitivity analysis of large models , 2007, Environ. Model. Softw..

[65]  Stefano Marsili-Libelli.  Environmental Systems Analysis with MATLAB , 2016 .

[66]  Hoshin Vijai Gupta,et al.  Model identification for hydrological forecasting under uncertainty , 2005 .

[67]  Anthony J. Jakeman,et al.  A catchment moisture deficit module for the IHACRES rainfall-runoff model , 2004, Environ. Model. Softw..

[68]  Joseph R. Kasprzyk,et al.  Introductory overview: Optimization using evolutionary algorithms and other metaheuristics , 2019, Environ. Model. Softw..

[69]  J. D. Stigter,et al.  A fast algorithm to assess local structural identifiability , 2015, Autom..

[70]  Hoshin Vijai Gupta,et al.  The quantity and quality of information in hydrologic models , 2015 .

[71]  H. Künsch,et al.  Practical identifiability analysis of large environmental simulation models , 2001 .

[72]  R. Bellman,et al.  On structural identifiability , 1970 .

[73]  Peter A. Vanrolleghem,et al.  Identification of biodegradation models under model and data uncertainty , 1996 .

[74]  Qiqi Wang,et al.  Erratum: Active Subspace Methods in Theory and Practice: Applications to Kriging Surfaces , 2013, SIAM J. Sci. Comput..

[75]  H. Akaike.  A new look at the statistical model identification , 1974 .

[76]  R. Stouffer,et al.  Stationarity Is Dead: Whither Water Management? , 2008, Science.

[77]  Andrew M. Stuart,et al.  Inverse problems: A Bayesian perspective , 2010, Acta Numerica.

[78]  Joseph H. A. Guillaume,et al.  Practical identifiability analysis of environmental models. , 2014 .

[79]  Sergei S. Kucherenko,et al.  Derivative based global sensitivity measures and their link with global sensitivity indices , 2009, Math. Comput. Simul..

[80]  George E. P. Box,et al.  Empirical Model‐Building and Response Surfaces , 1988 .

[81]  Joseph H. A. Guillaume,et al.  Prediction under uncertainty as a boundary problem: A general formulation using Iterative Closed Question Modelling , 2015, Environ. Model. Softw..

[82]  Stefano Marsili-Libelli,et al.  PEAS: A toolbox to assess the accuracy of estimated parameters in environmental models , 2007, Environ. Model. Softw..

[83]  A. Jakeman,et al.  How much complexity is warranted in a rainfall‐runoff model? , 1993 .

[84]  Alex A. Gorodetsky,et al.  Gradient-based optimization for regression in the functional tensor-train format , 2018, J. Comput. Phys..

[85]  P. Young,et al.  Simplicity out of complexity in environmental modelling: Occam's razor revisited. , 1996 .

[86]  Anthony J. Jakeman,et al.  Selecting among five common modelling approaches for integrated environmental assessment and management , 2013, Environ. Model. Softw..

[87]  David Gorsich,et al.  Improving Identifiability in Model Calibration Using Multiple Responses , 2012 .

[88]  Avi Ostfeld,et al.  Evolutionary algorithms and other metaheuristics in water resources: Current status, research challenges and future directions , 2014, Environ. Model. Softw..

[89]  Tiangang Cui,et al.  Data‐driven model reduction for the Bayesian solution of inverse problems , 2014, 1403.4290.

[90]  Dongbin Xiu,et al.  The Wiener-Askey Polynomial Chaos for Stochastic Differential Equations , 2002, SIAM J. Sci. Comput..

[91]  J. Doherty,et al.  Calibration‐constrained Monte Carlo analysis of highly parameterized models using subspace techniques , 2009 .

[92]  H. Gupta,et al.  A new framework for comprehensive, robust, and efficient global sensitivity analysis: 2. Application , 2016 .

[93]  D. M. Ely,et al.  A method for evaluating the importance of system state observations to model predictions, with application to the Death Valley regional groundwater flow system , 2004 .

[94]  G. Thomson.  The proof or disproof of the existence of general ability , 1919 .

[95]  Eva Balsa-Canto,et al.  AMIGO, a toolbox for advanced model identification in systems biology using global optimization , 2011, Bioinform..

[96]  I. Sobola,et al.  Global sensitivity indices for nonlinear mathematical models and their Monte Carlo estimates , 2001 .

[97]  Khachik Sargsyan,et al.  Enhancing ℓ1-minimization estimates of polynomial chaos expansions using basis selection , 2014, J. Comput. Phys..

[98]  Christian D Langevin,et al.  Quantifying Data Worth Toward Reducing Predictive Uncertainty , 2010, Ground water.

[99]  Mary C. Hill,et al.  Parameterization, sensitivity analysis, and inversion: an investigation using groundwater modeling of the surface-mined Tivoli-Guidonia basin (Metropolitan City of Rome, Italy) , 2016, Hydrogeology Journal.

[100]  Jim W. Hall,et al.  Sensitivity analysis of environmental models: A systematic review with practical workflow , 2014, Environ. Model. Softw..

[101]  S. Sorooshian,et al.  Effective and efficient global optimization for conceptual rainfall‐runoff models , 1992 .

[102]  A. Saltelli,et al.  Importance measures in global sensitivity analysis of nonlinear models , 1996 .

[103]  Lennart Ljung,et al.  On global identifiability for arbitrary model parametrizations , 1994, Autom..

[104]  Olivier Roustant,et al.  Calculations of Sobol indices for the Gaussian process metamodel , 2008, Reliab. Eng. Syst. Saf..

[105]  Anthony J. Jakeman,et al.  A review of surrogate models and their application to groundwater modeling , 2015 .

[106]  Soroosh Sorooshian,et al.  The Analysis of Structural Identifiability: Theory and Application to Conceptual Rainfall-Runoff Models , 1985 .

[107]  John Norton,et al.  An introduction to sensitivity assessment of simulation models , 2015, Environ. Model. Softw..

[108]  John Doherty,et al.  Two statistics for evaluating parameter identifiability and error reduction , 2009 .

[109]  S. Sorooshian,et al.  Uniqueness and observability of conceptual rainfall‐runoff model parameters: The percolation process examined , 1983 .

[110]  K. P. Sudheer,et al.  Methods used for the development of neural networks for the prediction of water resource variables in river systems: Current status and future directions , 2010, Environ. Model. Softw..

[111]  Keith R. Godfrey,et al.  Modal analysis of identifiablity of linear compartmental models , 1980 .

[112]  C. Cobelli,et al.  On parameter and structural identifiability: Nonunique observability/reconstructibility for identifiable systems, other ambiguities, and new definitions , 1980 .