Model validation: A bibliometric analysis of the literature

Validation is a crucial step in environmental and economic modeling that establishes the reliability of models to be used in decision-making contexts. It is often said that validation approaches proposed in the literature are not widely adopted, and different modeling fields do not benefit from each other. This study analyses a broad academic literature on model validation, mainly in environmental and decision sciences, by using an innovative combination of bibliometric and text-mining tools. The results show that a data-driven validation practice is prevalent. Although most publications in the studied dataset resemble each other, the most-cited ones tend to be different from the rest in terms of their abstracts’ content. Furthermore, the validation practices in different modeling areas are distinct, and do not extensively cite each other. In future, validation approaches can extend beyond data-oriented reliability for a wider acceptance of modeling in decision-making, and can synthesize the methods and views from various fields.

[1]  Michael I. Jordan,et al.  Latent Dirichlet Allocation , 2001, J. Mach. Learn. Res..

[2]  Ayalew Kassahun,et al.  Quality assurance in model based water management - review of existing practice and outline of new approaches , 2005, Environ. Model. Softw..

[3]  Daniel G. Brown,et al.  A review of current calibration and validation practices in land-change modeling , 2016, Environ. Model. Softw..

[4]  Stewart Robinson,et al.  Model Validation and Testing in Simulation: a Literature Review , 2016, SCOR.

[5]  Michael Obersteiner,et al.  Practice and perspectives in the validation of resource management models , 2018, Nature Communications.

[6]  N Oreskes,et al.  Verification, Validation, and Confirmation of Numerical Models in the Earth Sciences , 1994, Science.

[7]  Jeroen P. van der Sluijs,et al.  A way out of the credibility crisis of models used in integrated environmental assessment , 2002 .

[8]  Robert N. Broadus Toward a definition of “bibliometrics” , 1987, Scientometrics.

[9]  Mike Rivington,et al.  Validation of biophysical models: issues and methodologies. A review , 2011, Agronomy for Sustainable Development.

[10]  Joseph H. A. Guillaume,et al.  Characterising performance of environmental models , 2013, Environ. Model. Softw..

[11]  Silvio Funtowicz,et al.  Application of a checklist for quality assistance in environmental modelling to an energy model , 2005 .

[12]  L. Shawn Matott,et al.  Evaluating uncertainty in integrated environmental models: A review of concepts and tools , 2009 .

[13]  Mark Maslin,et al.  Uncertainty: Climate models at their limit? , 2012, Nature.

[14]  Volker Grimm,et al.  Merging validation and evaluation of ecological models to ‘evaludation’: A review of terminology and a practical approach , 2014 .

[15]  Geoffrey E. Hinton,et al.  Visualizing Data using t-SNE , 2008 .

[16]  Steven Bird,et al.  NLTK: The Natural Language Toolkit , 2002, ACL.

[17]  Jean-Louis Malouin,et al.  Model validation in operations research , 1983 .

[18]  Witold Pedrycz,et al.  Fifty years of Information Sciences: A bibliometric overview , 2018, Inf. Sci..

[19]  Jan H. Kwakkel,et al.  Analytics and Tech Mining for Engineering Managers , 2016 .

[20]  Linda Pilkey-Jarvis,et al.  :Useless Arithmetic: Why Environmental Scientists Cant Predict the Future , 2007 .

[21]  Silvio Funtowicz,et al.  Are all models wrong? , 2020, Computational and systems oncology.

[22]  Arthur C. Petersen,et al.  Variations on reliability: Connecting climate predictions to climate policy , 2013 .

[23]  Valeria Jana Schwanitz,et al.  Evaluating integrated assessment models of global climate change , 2013, Environ. Model. Softw..

[24]  Laurens van der Maaten,et al.  Accelerating t-SNE using tree-based algorithms , 2014, J. Mach. Learn. Res..

[25]  Osman Balci Validation, verification, and testing techniques throughout the life cycle of a simulation study , 1994, WSC '94.

[26]  Jerome R. Ravetz,et al.  Exploring the quality of evidence for complex and contested policy decisions , 2008 .

[27]  Emanuele Borgonovo,et al.  Forty years of the European Journal of Operational Research: A bibliometric overview , 2017, Eur. J. Oper. Res..

[28]  K. Beven Towards a coherent philosophy for modelling the environment , 2002, Proceedings of the Royal Society of London. Series A: Mathematical, Physical and Engineering Sciences.

[29]  Ludo Waltman,et al.  Software survey: VOSviewer, a computer program for bibliometric mapping , 2009, Scientometrics.

[30]  José M. Merigó,et al.  Forty years of Computers & Industrial Engineering: A bibliometric analysis , 2017, Comput. Ind. Eng..

[31]  Yaman Barlas,et al.  Formal aspects of model validity and validation in system dynamics , 1996 .

[32]  Anthony J. Jakeman,et al.  Ten iterative steps in development and evaluation of environmental models , 2006, Environ. Model. Softw..