Most computational hydrology is not reproducible, so is it really science?

Reproducibility is a foundational principle in scientific research. Yet in computational hydrology the code and data that actually produces published results are not regularly made available, inhibiting the ability of the community to reproduce and verify previous findings. In order to overcome this problem we recommend that reuseable code and formal workflows, which unambiguously reproduce published scientific results, are made available for the community alongside data, so that we can verify previous findings, and build directly from previous work. In cases where reproducing large-scale hydrologic studies is computationally very expensive and time-consuming, new processes are required to ensure scientific rigor. Such changes will strongly improve the transparency of hydrological research, and thus provide a more credible foundation for scientific advancement and policy support.

[1]  M. Kendall,et al.  The Logic of Scientific Discovery. , 1959 .

[2]  Günter Blöschl,et al.  On the future of journal publications in hydrology , 2014 .

[3]  Brian L. McGlynn,et al.  A review of the evolving perceptual model of hillslope flowpaths at the Maimai catchments, New Zealand , 2002 .

[4]  A. Casadevall,et al.  Misconduct accounts for the majority of retracted scientific publications , 2012, Proceedings of the National Academy of Sciences.

[5]  S. Sorooshian,et al.  Shuffled complex evolution approach for effective and efficient global minimization , 1993 .

[6]  Robert Gentleman,et al.  Statistical Analyses and Reproducible Research , 2007 .

[7]  Jeffery S. Horsburgh,et al.  Development of a Community Hydrologic Information System , 2009 .

[8]  Neil Malhotra,et al.  Publication Bias in the Social Sciences , 2014 .

[9]  Martyn P. Clark,et al.  Framework for Understanding Structural Errors (FUSE): A modular framework to diagnose differences between hydrological models , 2008 .

[10]  F. Collins,et al.  Policy: NIH plans to enhance reproducibility , 2014, Nature.

[11]  Martyn P. Clark,et al.  Development of a large-sample watershed-scale hydrometeorological data set for the contiguous USA: data set characteristics and assessment of regional variability in hydrologic model performance , 2014 .

[12]  Francesca Pianosi,et al.  A Matlab toolbox for Global Sensitivity Analysis , 2015, Environ. Model. Softw..

[13]  Brian A. Nosek,et al.  Promoting an open research culture , 2015, Science.

[14]  Reagan Moore,et al.  Using a data grid to automate data preparation pipelines required for regional-scale hydrologic modeling , 2016, Environ. Model. Softw..

[15]  Keith Beven,et al.  Multivariate seasonal period model rejection within the generalised likelihood uncertainty estimation procedure. , 2013 .

[16]  Christopher J. Duffy,et al.  Essential Terrestrial Variable data workflows for distributed water resources modeling , 2013, Environ. Model. Softw..

[17]  Patrick M. Reed,et al.  When are multiobjective calibration trade‐offs in hydrologic models meaningful? , 2012 .

[18]  E. Wood,et al.  Projected changes in drought occurrence under future global warming from multi-model, multi-scenario, IPCC AR4 simulations , 2008 .

[19]  Venkatesh Merwade,et al.  Enhancing the T-shaped learning profile when teaching hydrology using data, modeling, and visualization activities , 2015 .

[20]  Jonathan M. Borwein,et al.  Facilitating Reproducibility in Scientific Computing: Principles and Practice , 2015 .

[21]  J. Ioannidis,et al.  Replication validity of genetic association studies , 2001, Nature Genetics.

[22]  Neil Malhotra,et al.  Publication bias in the social sciences: Unlocking the file drawer , 2014, Science.

[23]  Keith Beven,et al.  Do we need a Community Hydrological Model? , 2015 .

[24]  Brooks Hanson,et al.  AGU's Data Policy: History and Context , 2014 .

[25]  Brian A. Nosek,et al.  Scientific Utopia , 2012, Perspectives on psychological science : a journal of the Association for Psychological Science.

[26]  Q. Duana,et al.  Model Parameter Estimation Experiment (MOPEX): An overview of science strategy and major results from the second and third workshops , 2006 .

[27]  Yuxin Ma,et al.  HydroViz: design and evaluation of a Web-based tool for improving hydrology education , 2012 .

[28]  Dmitri Kavetski,et al.  Towards more systematic perceptual model development: a case study using 3 Luxembourgish catchments , 2015 .

[29]  R. Woods,et al.  Catchment Classification and Hydrologic Similarity , 2006, Geography Compass.

[30]  Hubert H. G. Savenije,et al.  Joint editorial – Fostering innovation and improving impact assessment for journal publications in hydrology , 2016 .

[31]  Brian A. Nosek,et al.  An Open, Large-Scale, Collaborative Effort to Estimate the Reproducibility of Psychological Science , 2012, Perspectives on psychological science : a journal of the Association for Psychological Science.

[32]  Peter A. Troch,et al.  Land surface hydrology. , 2008 .

[33]  David R. Maidment.  Bringing Water Data Together , 2008 .

[34]  Jasper A. Vrugt,et al.  Reproducible Research in Vadose Zone Sciences , 2015 .

[35]  Yolanda Gil,et al.  OntoSoft: Capturing Scientific Software Metadata , 2015, K-CAP.

[36]  David L. Donoho,et al.  WaveLab and Reproducible Research , 1995 .

[37]  J. Ioannidis.  Why Most Published Research Findings Are False , 2005, PLoS medicine.

[38]  J. Ioannidis.  Why Most Published Research Findings Are False , 2019, CHANCE.

[39]  Jeffery S. Horsburgh,et al.  HydroShare: Sharing Diverse Environmental Data Types and Models as Social Objects with Application to the Hydrology Domain , 2016 .

[40]  Venkatesh Merwade,et al.  Moving university hydrology education forward with community-based geoinformatics, data and modeling resources , 2012 .

[41]  Olaf Kolditz,et al.  Surface‐subsurface model intercomparison: A first set of benchmark results to diagnose integrated hydrology and feedbacks , 2014 .

[42]  Keith Beven,et al.  The future of distributed models: model calibration and uncertainty prediction. , 1992 .

[43]  Leonard J. Lane,et al.  Hydraulic Roughness Coefficients for Native Rangelands , 1992 .

[44]  Eric F. Wood,et al.  A land-surface hydrology parameterization with subgrid variability for general circulation models , 1992 .

[45]  John Wainwright,et al.  Resistance to overland flow on semiarid grassland and shrubland hillslopes, Walnut Gulch, southern Arizona , 1994 .

[46]  F. Prinz,et al.  Believe it or not: how much can we rely on published data on potential drug targets? , 2011, Nature Reviews Drug Discovery.

[47]  Christopher Hutton,et al.  How significant (p < 0.05) is geomorphic research? , 2014 .

[48]  C. Luce.  Runoff Prediction in Ungauged Basins: Synthesis Across Processes, Places and Scales , 2014 .

[49]  Nicholas Smith,et al.  Climategate, Public Opinion, and the Loss of Trust , 2010, SSRN Electronic Journal.

[50]  P. J. Smith,et al.  A novel framework for discharge uncertainty quantification applied to 500 UK gauging stations , 2015, Water resources research.

[51]  John Wainwright,et al.  On determining resistance to interrill overland flow , 1994 .

[52]  J. Freer,et al.  Benchmarking observational uncertainties for hydrology: rainfall, river discharge and water quality , 2012 .

[53]  Martyn P. Clark,et al.  Improving the theoretical underpinnings of process-based hydrologic models , 2016 .

[54]  Hubert H. G. Savenije,et al.  Joint editorial: Fostering innovation and improving impact assessment for journal publications in hydrology , 2016, Hydrology and Earth System Sciences.

[55]  J. Goodall,et al.  An ontology for component‐based models of water resource systems , 2013 .

[56]  Mary C. Whitton,et al.  Server‐side workflow execution using data grid technology for reproducible analyses of data‐intensive hydrologic systems , 2016 .

[57]  R. Peng.  Reproducible Research in Computational Science , 2011, Science.

[58]  Berit Arheimer,et al.  Large-scale hydrological modelling by using modified PUB recommendations: the India-HYPE case , 2015 .

[59]  M. Jennions,et al.  Relationships fade with time: a meta-analysis of temporal trends in publication in ecology and evolution , 2002, Proceedings of the Royal Society of London. Series B: Biological Sciences.

[60]  Göran Lindström,et al.  Virtual laboratories: new opportunities for collaborative water science , 2014, Hydrology and Earth System Sciences.

[61]  Alva L. Couch,et al.  HydroShare: Advancing Collaboration through Hydrologic Data and Model Sharing , 2015 .

[62]  S. Sorooshian,et al.  A Shuffled Complex Evolution Metropolis algorithm for optimization and uncertainty assessment of hydrologic model parameters , 2002 .

[63]  A. W. Harbaugh.  MODFLOW-2005 : the U.S. Geological Survey modular ground-water model--the ground-water flow process , 2005 .

[64]  Gordon S. Blair,et al.  Heads in the cloud: innovation in data and model dissemination , 2014 .

[65]  MalekiArian,et al.  Reproducible Research in Computational Harmonic Analysis , 2009 .

[66]  C. Begley,et al.  Drug development: Raise standards for preclinical cancer research , 2012, Nature.

[67]  Thorsten Wagener,et al.  TOOLS FOR TEACHING HYDROLOGICAL AND ENVIRONMENTAL MODELING , 2007 .

[68]  Arian Maleki,et al.  Reproducible Research in Computational Harmonic Analysis , 2009, Computing in Science & Engineering.

[69]  K. Beven,et al.  A physically based, variable contributing area model of basin hydrology , 1979 .

[70]  Murugesu Sivapalan,et al.  Dominant flood generating mechanisms across the United States , 2016 .

[71]  Ramesh S. V. Teegavarapu,et al.  Estimation of missing precipitation records integrating surface interpolation techniques and spatio-temporal association rules , 2009 .

[72]  B. Arheimer,et al.  Development and testing of the HYPE (Hydrological Predictions for the Environment) water quality model for different spatial scales , 2010 .

[73]  Jill P Mesirov,et al.  Accessible Reproducible Research , 2010, Science.

[74]  J. Famiglietti,et al.  A decade of RAPID—Reflections on the development of an open source geoscience code , 2016 .

[75]  Kenneth G. Renard,et al.  A brief background on the U.S. Department of Agriculture Agricultural Research Service Walnut Gulch Experimental Watershed , 2008 .

[76]  Keith Beven,et al.  Uniqueness of place and process representations in hydrological modelling , 2000 .

[77]  Malcolm Newson,et al.  Plynlimon research: The first two decades , 1991 .