Batch sequential design of optimal experiments for improved predictive maturity in physics-based modeling

The focus of nuclear fuel design and maintenance has been shifting from a primarily empirical endeavor to that of highly advanced simulations characterized by experiment-based calibration, validation and uncertainty quantification. The experimental data available for calibration and validation, however, is limited by the availability of resources. This limitation poses difficulties especially if the model is to be executed to predict at different settings and/or regimes within a domain. To assure that the model exhibits predictive maturity throughout the domain of applicability, a sufficient number of experiments must be conducted to explore this domain. Given the limited resources, it is therefore crucial to design validation experiments to maximize the improvement in the predictive capability of the physics-based numerical models. This article contributes to the recent developments in the optimal design of validation experiments by evaluating the performance of several experiment selection criteria that specify the specific benefits desired from future experiments. Our focus is on the Batch Sequential Design methods, which for a given set of initial experiments and a selection criteria, iteratively select a batch of future experiments. The performance of various selection criteria in improving model predictiveness are compared considering not only the empirically identified model discrepancy, but also the coverage of the domain of applicability. The manuscript provides an extensive simulation-based study on a polycrystal plasticity material model, utilizing an established index that quantifies predictive maturity of numerical models.

[1]  Runchu Zhang,et al.  Theory of optimal blocking of nonregular factorial designs , 2004 .

[2]  François M. Hemez,et al.  The dangers of sparse sampling for the quantification of margin and uncertainty , 2011, Reliab. Eng. Syst. Saf..

[3]  T. J. Mitchell,et al.  Exploratory designs for computational experiments , 1995 .

[4]  C. S. Hartley,et al.  Modeling the mechanical response of polycrystals deforming by climb and glide , 2010 .

[5]  Xiang Jiang,et al.  Bayesian cross-entropy methodology for optimal design of validation experiments , 2006 .

[6]  Jason L. Loeppky,et al.  Batch sequential designs for computer experiments , 2010 .

[7]  M. Olsson,et al.  Efficient design of experiments for structural optimization using significance screening , 2012 .

[8]  Osman Balci,et al.  A collaborative evaluation environment for credibility assessment of modeling and simulation applications , 2002, Proceedings of the Winter Simulation Conference.

[9]  R. A. Lebensohn,et al.  A Resource Allocation Framework for Experiment-Based Validation of Numerical Models , 2015 .

[10]  Ismail Farajpour,et al.  Error and Uncertainty Analysis of Inexact and Imprecise Computer Models , 2013, J. Comput. Civ. Eng..

[11]  Sallie Keller-McNulty,et al.  Combining experimental data and computer simulations, with an application to flyer plate experiments , 2006 .

[12]  Thomas J. Santner,et al.  The Design and Analysis of Computer Experiments , 2003, Springer Series in Statistics.

[13]  David Lindley,et al.  Bayesian Statistics, a Review , 1987 .

[14]  Tiefu Shao Toward a structured approach to simulation-based engineering design under uncertainty , 2007 .

[15]  A. O'Hagan,et al.  Bayesian calibration of computer models , 2001 .

[16]  G. Box,et al.  A Basis for the Selection of a Response Surface Design , 1959 .

[17]  Robert Rosner Making Nuclear Energy Work , 2008 .

[18]  Gordon Graham,et al.  The use of a modified Fedorov exchange algorithm to optimise sampling times for population pharmacokinetic experiments , 2005, Comput. Methods Programs Biomed..

[19]  Jason L. Loeppky,et al.  Batch sequential design to achieve predictive maturity with calibrated computer models , 2011, Reliab. Eng. Syst. Saf..

[20]  William I. Notz,et al.  Sequential adaptive designs in computer experiments for response surface model fit , 2008 .

[21]  Laura Painton Swiler,et al.  Calibration, validation, and sensitivity analysis: What's what , 2006, Reliab. Eng. Syst. Saf..

[22]  Shapour Azarm,et al.  An accumulative error based adaptive design of experiments for offline metamodeling , 2009 .

[23]  R. D. Cook,et al.  A Comparison of Algorithms for Constructing Exact D-Optimal Designs , 1980 .

[24]  K. R. Shah,et al.  Theory of Optimal Designs , 1989 .

[25]  H. MyersR.,et al.  Response surface methodology: 19661988 , 1989 .

[26]  Fred van Keulen,et al.  Gradient-enhanced response surface building , 2002 .

[27]  G. Rennen,et al.  Nested maximin Latin hypercube designs , 2009 .

[28]  François M. Hemez,et al.  Improved best estimate plus uncertainty methodology, including advanced validation concepts, to license evolving nuclear reactors , 2011 .

[29]  P. James McLellan,et al.  Design of Optimal Sequential Experiments to Improve Model Predictions from a Polyethylene Molecular Weight Distribution Model , 2010 .

[30]  Henry P. Wynn,et al.  Maximum entropy sampling , 1987 .

[31]  Werner G. Müller,et al.  batch sequential design for a nonlinear estimation problem , 1989 .

[32]  André I. Khuri,et al.  Response surface methodology: 1966–1988 , 1989 .

[33]  D. Higdon,et al.  Computer Model Calibration Using High-Dimensional Output , 2008 .

[34]  Toshimasa Yoshiie,et al.  Factors That Influence Cascade-Induced Defect Growth in Pure Metals and Model Alloys , 2005 .

[35]  James O. Berger,et al.  Markov chain Monte Carlo-based approaches for inference in computationally intensive inverse problems , 2003 .

[36]  Max D. Morris,et al.  Minimax and Maximin Distance Designs for Computer Experiments with Derivative Information , 2011 .

[37]  François M. Hemez,et al.  Predictive Maturity of Computer Models Using Functional and Multivariate Output , 2008 .

[38]  David Draper,et al.  Assessment and Propagation of Model Uncertainty , 2011 .

[39]  François M. Hemez,et al.  Defining predictive maturity for validated numerical simulations , 2010 .

[40]  Ram K. Tripathi,et al.  An Uncertainty Structure Matrix for Models and Simulations , 2008 .

[41]  Henry P. Wynn,et al.  [Design and Analysis of Computer Experiments]: Rejoinder , 1989 .

[42]  Dursun A. Bulutoglu,et al.  D-optimal and near D-optimal 2k fractional factorial designs of resolution V , 2009 .

[43]  François M. Hemez,et al.  Defining Coverage of a Domain Using a Modified Nearest-Neighbor Metric , 2013 .

[44]  N. Metropolis,et al.  Equation of State Calculations by Fast Computing Machines , 1953, Resonance.

[45]  Timothy G. Trucano,et al.  Predictive Capability Maturity Model for computational modeling and simulation. , 2007 .

[46]  David Shropshire,et al.  VISION: Verifiable Fuel Cycle Simulation Model , 2009 .

[47]  A. R. Manson,et al.  Optimal Experimental Designs in Two Dimensions Using Minimum Bias Estimation , 1978 .

[48]  François M. Hemez,et al.  A forecasting metric for predictive modeling , 2011 .