Failure analysis of parameter-induced simulation crashes in climate models

Abstract. Simulations using IPCC (Intergovernmental Panel on Climate Change)-class climate models are subject to fail or crash for a variety of reasons. Quantitative analysis of the failures can yield useful insights to better understand and improve the models. During the course of uncertainty quantification (UQ) ensemble simulations to assess the effects of ocean model parameter uncertainties on climate simulations, we experienced a series of simulation crashes within the Parallel Ocean Program (POP2) component of the Community Climate System Model (CCSM4). About 8.5% of our CCSM4 simulations failed for numerical reasons at combinations of POP2 parameter values. We applied support vector machine (SVM) classification from machine learning to quantify and predict the probability of failure as a function of the values of 18 POP2 parameters. A committee of SVM classifiers readily predicted model failures in an independent validation ensemble, as assessed by the area under the receiver operating characteristic (ROC) curve metric (AUC > 0.96). The causes of the simulation failures were determined through a global sensitivity analysis. Combinations of 8 parameters related to ocean mixing and viscosity from three different POP2 parameterizations were the major sources of the failures. This information can be used to improve POP2 and CCSM4 by incorporating correlations across the relevant parameters. Our method can also be used to quantify, predict, and understand simulation crashes in other complex geoscientific models.

[1]  Steve M. Easterbrook,et al.  Guest Editors' Introduction: Climate Change - Science and Software , 2011, IEEE Software.

[2]  David A. Ham,et al.  Automated continuous verification for numerical simulation , 2011 .

[3]  S. Bates,et al.  The CCSM4 Ocean Component , 2012 .

[4]  Christopher M. Bishop,et al.  Pattern Recognition and Machine Learning (Information Science and Statistics) , 2006 .

[5]  Gustavo E. A. P. A. Batista,et al.  Class Imbalances versus Class Overlapping: An Analysis of a Learning System Behavior , 2004, MICAI.

[6]  Spencer Rugaber,et al.  Managing Software Complexity and Variability in Coupled Climate Models , 2011, IEEE Software.

[7]  P. Gent,et al.  Isopycnal mixing in ocean circulation models , 1990 .

[8]  Thomas G. Dietterich.  Multiple Classifier Systems , 2000, Lecture Notes in Computer Science.

[9]  James D. Annan,et al.  Parameter estimation in an intermediate complexity earth system model using an ensemble Kalman filter , 2005 .

[10]  I. Sobol.  Global sensitivity indices for nonlinear mathematical models and their Monte Carlo estimates , 2001 .

[11]  Jon Pipitone,et al.  Assessing climate model software quality: a defect density analysis of three models , 2012 .

[12]  Daniel Kroening,et al.  A Survey of Automated Techniques for Formal Software Verification , 2008, IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems.

[13]  C. Marzban.  The ROC Curve and the Area under It as Performance Measures , 2004 .

[14]  Richard J. Beckman,et al.  A Comparison of Three Methods for Selecting Values of Input Variables in the Analysis of Output From a Computer Code , 2000, Technometrics.

[15]  M. Stein.  Large sample properties of simulations using latin hypercube sampling , 1987 .

[16]  Michael Schulz,et al.  The Community Climate System Model (version 2.0.1) on the HLRN supercomputer: Adjustment, accelerated integration, and present-day control run , 2006 .

[17]  Leo Breiman,et al.  Random Forests , 2001, Machine Learning.

[18]  W. Washington,et al.  An Introduction to Three-Dimensional Climate Modeling , 1986 .

[19]  Donald D. Lucas,et al.  Poster: Data intensive uncertainty quantification: applications to climate modeling , 2011, SC '11 Companion.

[20]  W. Large,et al.  Oceanic vertical mixing: a review and a model with a nonlocal boundary layer parameterization , 1994 .

[21]  Marika M. Holland,et al.  Ocean viscosity and climate , 2008 .

[22]  Frank O. Bryan,et al.  Equatorial Circulation of a Global Ocean Climate Model with Anisotropic Horizontal Viscosity , 2001 .

[23]  Steve M. Easterbrook,et al.  Climate change: a grand software challenge , 2010, FoSER '10.

[24]  Bruno Sudret,et al.  Global sensitivity analysis using polynomial chaos expansions , 2008, Reliab. Eng. Syst. Saf..

[25]  Mrinal K. Sen,et al.  Error Reduction and Convergence in Climate Prediction , 2008 .

[26]  J A Swets,et al.  Measuring the accuracy of diagnostic systems. , 1988, Science.

[27]  Stephen G. Yeager,et al.  The global climatology of an interannually varying air–sea flux data set , 2009 .

[28]  M. Webb,et al.  Quantification of modelling uncertainties in a large ensemble of climate change simulations , 2004, Nature.

[29]  Pierre Morel.  An Introduction to Three‐Dimensional Climate Modeling , 1988 .

[30]  Tom Fawcett,et al.  An introduction to ROC analysis , 2006, Pattern Recognit. Lett..

[31]  Tan Yee Fan,et al.  A Tutorial on Support Vector Machine , 2009 .

[32]  Menner A. Tatang,et al.  An efficient method for parametric uncertainty analysis of numerical geophysical models , 1997 .

[33]  Leo Breiman,et al.  Bagging Predictors , 1996, Machine Learning.

[34]  Andrei P. Sokolov,et al.  Estimating Probability Distributions from Complex Models with Bifurcations: The Case of Ocean Circulation Collapse , 2007 .

[35]  G. Danabasoglu,et al.  The Community Climate System Model Version 4 , 2004 .

[36]  D. Hosmer,et al.  Applied Logistic Regression , 1991 .

[37]  Chih-Jen Lin,et al.  LIBSVM: A library for support vector machines , 2011, TIST.

[38]  N. Wiener.  The Homogeneous Chaos , 1938 .

[39]  Vladimir N. Vapnik,et al.  The Nature of Statistical Learning Theory , 2000, Statistics for Engineering and Information Science.

[40]  Sotiris B. Kotsiantis,et al.  Supervised Machine Learning: A Review of Classification Techniques , 2007, Informatica.

[41]  Donald D. Lucas,et al.  Parametric sensitivity and uncertainty analysis of dimethylsulfide oxidation in the clear-sky remote marine boundary layer , 2004 .

[42]  Christopher J. C. Burges,et al.  A Tutorial on Support Vector Machines for Pattern Recognition , 1998, Data Mining and Knowledge Discovery.

[43]  Joseph A. C. Delaney.  Sensitivity analysis , 2018, The African Continental Free Trade Area: Economic and Distributional Effects.

[44]  Sylvain Arlot,et al.  A survey of cross-validation procedures for model selection , 2009, 0907.4728.

[45]  Jon C. Helton,et al.  Survey of sampling-based methods for uncertainty and sensitivity analysis , 2006, Reliab. Eng. Syst. Saf..

[46]  Nasser M. Nasrabadi,et al.  Pattern Recognition and Machine Learning , 2006, Technometrics.

[47]  Leo Breiman,et al.  Classification and Regression Trees , 1984 .

[48]  Vladimir Vapnik,et al.  The Nature of Statistical Learning , 1995 .

[49]  John Platt,et al.  Probabilistic Outputs for Support vector Machines and Comparisons to Regularized Likelihood Methods , 1999 .

[50]  I. Sobola,et al.  Global sensitivity indices for nonlinear mathematical models and their Monte Carlo estimates , 2001 .

[51]  Leonard A. Smith,et al.  Uncertainty in predictions of the climate response to rising levels of greenhouse gases , 2005, Nature.

[52]  Wei-Yin Loh,et al.  Classification and regression trees , 2011, WIREs Data Mining Knowl. Discov..

[53]  Mariana Vertenstein,et al.  The Parallel Ocean Program (POP) reference manual: Ocean component of the Community Climate System Model (CCSM) , 2010 .

[54]  Benjamin M. Sanderson,et al.  A Multimodel Study of Parametric Uncertainty in Predictions of Climate Response to Rising Greenhouse Gas Concentrations , 2011 .

[55]  B. Fox‐Kemper,et al.  Parameterization of Mixed Layer Eddies. Part I. Theory and Diagnosis , 2008 .

[56]  Andy J. Keane,et al.  Engineering Design via Surrogate Modelling - A Practical Guide , 2008 .

[57]  D. Stensrud.  Parameterization Schemes: Keys to Understanding Numerical Weather Prediction Models , 2007 .

[58]  S. Jayne,et al.  The Impact of Abyssal Mixing Parameterizations in an Ocean General Circulation Model , 2009 .

[59]  S. Emori,et al.  Perturbed physics ensemble using the MIROC5 coupled atmosphere–ocean GCM without flux corrections: experimental design and results , 2012, Climate Dynamics.

[60]  Corinna Cortes,et al.  Support-Vector Networks , 1995, Machine Learning.

[61]  Baylor Fox-Kemper,et al.  Parameterization of Mixed Layer Eddies. I: Theory and Diagnosis , 2007 .

[62]  David W. Hosmer,et al.  Applied Logistic Regression , 1991 .

[63]  Dongbin Xiu,et al.  The Wiener-Askey Polynomial Chaos for Stochastic Differential Equations , 2002, SIAM J. Sci. Comput..

[64]  C. Jones,et al.  The HadGEM2 family of Met Office Unified Model climate configurations , 2011 .

[65]  Stephen M. Griffies,et al.  Fundamentals of Ocean Climate Models , 2004 .

[66]  James C. McWilliams,et al.  Anisotropic horizontal viscosity for ocean models , 2003 .

[67]  Thomas L. Clune,et al.  Software Testing and Verification in Climate Model Development , 2011, IEEE Software.

[68]  J. Rougier,et al.  Precalibrating an intermediate complexity climate model , 2018 .

[69]  Steve M. Easterbrook,et al.  Engineering the Software for Understanding Climate Change , 2009, Computing in Science & Engineering.

[70]  D. Randall,et al.  Climate models and their evaluation , 2007 .

[71]  Wei Chen,et al.  Assessing the Reliability of Complex Models: Mathematical and Statistical Foundations of Verification, Validation, and Uncertainty Quantification , 2012 .