Tuning without over-tuning: parametric uncertainty quantification for the NEMO ocean model

Abstract. In this paper we discuss climate model tuning and present an iterative automatic tuning method from the statistical science literature. The method, which we refer to here as iterative refocussing (though also known as history matching), avoids many of the common pitfalls of automatic tuning procedures that are based on optimisation of a cost function, principally the over-tuning of a climate model due to using only partial observations. This avoidance comes by seeking to rule out parameter choices that we are confident could not reproduce the observations, rather than seeking the model that is closest to them (a procedure that risks over-tuning). We comment on the state of climate model tuning and illustrate our approach through three waves of iterative refocussing of the NEMO (Nucleus for European Modelling of the Ocean) ORCA2 global ocean model run at 2° resolution. We show how at certain depths the anomalies of global mean temperature and salinity in a standard configuration of the model exceeds 10 standard deviations away from observations and show the extent to which this can be alleviated by iterative refocussing without compromising model performance spatially. We show how model improvements can be achieved by simultaneously perturbing multiple parameters, and illustrate the potential of using low-resolution ensembles to tune NEMO ORCA configurations at higher resolutions.

[1]  Gustavo Goni,et al.  Measuring the Atlantic Meridional Overturning Circulation , 2015 .

[2]  Timothy P. Boyer,et al.  World ocean atlas 2013. Volume 2, Salinity , 2002 .

[3]  T. J. Mitchell,et al.  Exploratory designs for computational experiments , 1995 .

[4]  Frank O. Bryan,et al.  Coordinated Ocean-ice Reference Experiments (COREs) , 2009 .

[5]  Bengamin I. Moat,et al.  Measuring the Atlantic Meridional Overturning Circulation at 26°N , 2015 .

[6]  Michael Goldstein,et al.  Small Sample Bayesian Designs for Complex High-Dimensional Models Based on Information Gained Using Fast Approximations , 2009, Technometrics.

[7]  D. Klocke,et al.  Tuning the climate of a global model , 2012 .

[8]  Ian Vernon,et al.  Efficient uniform designs for multi-wave computer experiments , 2013, 1309.3520.

[9]  Feng Xie,et al.  An automatic and effective parameter optimization method for model tuning , 2015 .

[10]  M. J. Bayarri,et al.  Computer model validation with functional output , 2007, 0711.3271.

[11]  A. O'Hagan,et al.  Bayesian calibration of computer models , 2001 .

[12]  William E. Johns,et al.  Temporal Variability of the Atlantic Meridional Overturning Circulation at 26.5°N , 2007, Science.

[13]  Jenný Brynjarsdóttir,et al.  Learning about physical parameters: the importance of model discrepancy , 2014 .

[14]  John Siddorn,et al.  GO5.0: The joint NERC-Met Office NEMO global ocean model for use in coupled and forced applications , 2013 .

[15]  G. Danabasoglu,et al.  The Community Climate System Model Version 4 , 2011 .

[16]  S. Bony,et al.  Climate change projections using the IPSL-CM5 Earth System Model: from CMIP3 to CMIP5 , 2013, Climate Dynamics.

[17]  Bruno Blanke,et al.  Water Mass Export from Drake Passage to the Atlantic, Indian, and Pacific Oceans: A Lagrangian Model Analysis , 2005 .

[18]  S. Levitus,et al.  World ocean atlas 2013. Volume 1, Temperature , 2002 .

[19]  E. Balopoulos,et al.  Objective analysis of temperature and salinity historical data set over the Mediterranean basin , 1998, IEEE Oceanic Engineering Society. OCEANS'98. Conference Proceedings (Cat. No.98CH36259).

[20]  Serge Guillas,et al.  Sequential Design with Mutual Information for Computer Experiments (MICE): Emulation of a Tsunami Model , 2014, SIAM/ASA J. Uncertain. Quantification.

[21]  Jeremy E. Oakley,et al.  A web-based tool for eliciting probability distributions from experts , 2014, Environ. Model. Softw..

[22]  F. Pukelsheim The Three Sigma Rule , 1994 .

[23]  Stephen G. Yeager,et al.  The global climatology of an interannually varying air–sea flux data set , 2009 .

[24]  Valérie Dulière,et al.  On the representation of high latitude processes in the ORCA-LIM global coupled sea ice–ocean model , 2005 .

[25]  N. Draper,et al.  Applied Regression Analysis: Draper/Applied Regression Analysis , 1998 .

[26]  Karl E. Taylor,et al.  An overview of CMIP5 and the experiment design , 2012 .

[27]  John F. B. Mitchell,et al.  THE WCRP CMIP3 Multimodel Dataset: A New Era in Climate Change Research , 2007 .

[28]  James M. Salter,et al.  Identifying and removing structural biases in climate models with history matching , 2015, Climate Dynamics.

[29]  Michael Goldstein,et al.  Fast linked analyses for scenario‐based hierarchies , 2012 .

[30]  Michael Goldstein,et al.  History matching for exploring and reducing climate model parameter space using observations and a large perturbed physics ensemble , 2013, Climate Dynamics.

[31]  Chris Harris,et al.  Design and implementation of the infrastructure of HadGEM3: the next-generation Met Office climate modelling system , 2010 .

[32]  Michael Steele,et al.  PHC: A Global Ocean Hydrography with a High-Quality Arctic Ocean , 2001 .

[33]  Loic Le Gratiet,et al.  Bayesian Analysis of Hierarchical Multifidelity Codes , 2011, SIAM/ASA J. Uncertain. Quantification.

[34]  Andrew Gettelman,et al.  The Art and Science of Climate Model Tuning , 2017 .

[35]  A. O'Hagan,et al.  Gaussian process emulation of dynamic computer codes , 2009 .

[36]  Jonathan M. Gregory,et al.  Calibrated prediction of Pine Island Glacier retreat during the 21st and 22nd centuries with a coupled flowline model , 2012 .

[37]  M. Huddleston,et al.  Quality control of ocean temperature and salinity profiles — Historical and real-time data , 2007 .

[38]  H. Douville,et al.  The CNRM-CM5.1 global climate model: description and basic evaluation , 2013, Climate Dynamics.

[39]  Stephen G. Yeager,et al.  Diurnal to decadal global forcing for ocean and sea-ice models: The data sets and flux climatologies , 2004 .

[40]  Nick Rayner,et al.  EN4: Quality controlled ocean temperature and salinity profiles and monthly objective analyses with uncertainty estimates , 2013 .

[41]  Elizabeth C. Kent,et al.  Global analyses of sea surface temperature, sea ice, and night marine air temperature since the late nineteenth century , 2003 .

[42]  Jason L. Loeppky,et al.  Batch sequential designs for computer experiments , 2010 .

[43]  Jonathan Rougier,et al.  Probabilistic Inference for Future Climate Using an Ensemble of Climate Model Evaluations , 2007 .

[44]  Sally A. McFarlane,et al.  Uncertainty quantification and parameter tuning in the CAM5 Zhang‐McFarlane convection scheme and impact of improved convection on the global circulation and climate , 2012 .

[45]  James M. Salter,et al.  A comparison of statistical emulation methodologies for multi‐wave calibration of environmental models , 2016, Environmetrics.

[46]  Michael Goldstein,et al.  Reified Bayesian modelling and inference for physical systems , 2009 .

[47]  Jonathan Rougier,et al.  Emulating the sensitivity of the HadSM 3 climate model using ensembles from different but related experiments , 2006 .

[48]  Michel Crucifix,et al.  The new hadley centre climate model (HadGEM1) : Evaluation of coupled simulations , 2006 .

[49]  Ian Vernon,et al.  Galaxy formation : a Bayesian uncertainty analysis. , 2010 .

[50]  John F. B. Mitchell,et al.  THE WCRP CMIP 3 MULTIMODEL DATASET A New Era in Climate Change Research , 2017 .

[51]  Michael A. West,et al.  A dynamic modelling strategy for Bayesian computer model emulation , 2009 .

[52]  James R. Gattiker,et al.  The potential of an observational data set for calibration of a computationally expensive computer model , 2013 .

[53]  Jonathan Rougier,et al.  Analyzing the Climate Sensitivity of the HadSM3 Climate Model Using Ensembles from Different but Related Experiments , 2009 .

[54]  Klaus Wyser,et al.  EC-Earth V2.2: description and validation of a new seamless earth system prediction model , 2012, Climate Dynamics.

[55]  A. O'Hagan,et al.  Predicting the output from a complex computer code when fast approximations are available , 2000 .

[56]  Bengamin I. Moat,et al.  Atlantic meridional overturning circulation observed by the RAPID-MOCHA-WBTS (RAPID-Meridional Overturning Circulation and Heatflux Array-Western Boundary Time Series) array at 26N from 2004 to 2015 , 2016 .

[57]  D. Williamson,et al.  Exploratory ensemble designs for environmental models using k-extended Latin Hypercubes , 2015, Environmetrics.

[58]  J. Rougier,et al.  Precalibrating an intermediate complexity climate model , 2018 .

[59]  Michael Schulz,et al.  The Community Climate System Model (version 2.0.1) on the HLRN supercomputer: Adjustment, accelerated integration, and present-day control run , 2006 .

[60]  Tianjun Zhou,et al.  Parameter Tuning and Calibration of RegCM3 with MIT–Emanuel Cumulus Parameterization Scheme over CORDEX East Asia Domain , 2014 .

[61]  Daniel B. Williamson,et al.  Evolving Bayesian Emulators for Structured Chaotic Time Series, with Application to Large Climate Models , 2014, SIAM/ASA J. Uncertain. Quantification.

[62]  M. Webb,et al.  Multivariate probabilistic projections using imperfect climate models part I: outline of methodology , 2012, Climate Dynamics.

[63]  A. Alessandri,et al.  INGV-CMCC Carbon (ICC): A Carbon Cycle Earth System Model , 2009 .

[64]  Matthew D. Collins,et al.  UK Climate Projections Science Report: Climate Change Projections , 2009 .

[65]  Richard F. Gunst,et al.  Applied Regression Analysis , 1999, Technometrics.

[66]  A. Seheult,et al.  Pressure Matching for Hydrocarbon Reservoirs: A Case Study in the Use of Bayes Linear Strategies for Large Computer Experiments , 1997 .

[67]  D. Higdon,et al.  Computer Model Calibration Using High-Dimensional Output , 2008 .