Controlled Morris method: A new factor screening approach empowered by a distribution-free sequential multiple testing procedure

Abstract The Morris method (MM) is a well known model-free factor screening approach that is considered particularly effective when the number of factors is large or when the computer model is computationally expensive to run. In this paper, we propose the controlled Morris method (CMM) for simulation-based factor screening, which integrates a distribution-free sequential multiple testing procedure with MM to control the Type I and Type II familywise error rates at the prescribed levels while achieving a high computational efficiency. Numerical experiments are provided to demonstrate the efficacy and efficiency of CMM.

[1]  Juan Miguel Ortiz-de-Lazcano-Lobato,et al.  Soft clustering for nonparametric probability density function estimation , 2007, Pattern Recognit. Lett..

[2]  Max D. Morris,et al.  Input screening: Finding the important model inputs on a budget , 2006, Reliab. Eng. Syst. Saf..

[3]  Bruce E. Ankenman,et al.  Screening for Dispersion Effects by Sequential Bifurcation , 2014, ACM Trans. Model. Comput. Simul..

[4]  Weichung Wang,et al.  Optimizing Two-Level Supersaturated Designs Using Swarm Intelligence Techniques , 2016, Technometrics.

[5]  Wen Shi,et al.  An Efficient Morris Method-Based Framework for Simulation Factor Screening , 2019, INFORMS J. Comput..

[6]  Shyamal K. De,et al.  Step-up and step-down methods for testing multiple hypotheses in sequential experiments , 2012 .

[7]  A. R. Ravishankara,et al.  A sensitivity analysis of key natural factors in the modeled global acetone budget , 2017 .

[8]  David B. Dunson,et al.  Statistics in the big data era: Failures of the machine , 2018 .

[9]  Biagio Ciuffo,et al.  Combining screening and metamodel-based methods: An efficient sequential approach for the sensitivity analysis of model outputs , 2015, Reliab. Eng. Syst. Saf..

[10]  Lee W. Schruben,et al.  An experimental procedure for simulation response surface model identification , 1987, CACM.

[11]  W. Härdle.  Applied Nonparametric Regression , 1991 .

[12]  Danijel Skocaj,et al.  Multivariate online kernel density estimation with Gaussian kernels , 2011, Pattern Recognit..

[13]  Hong Wan,et al.  Sequential procedures for multiple responses factor screening , 2014, Proceedings of the Winter Simulation Conference 2014.

[14]  Guohui Zhang,et al.  A Gaussian Kernel-Based Approach for Modeling Vehicle Headway Distributions , 2014, Transp. Sci..

[15]  Roger D. Braddock,et al.  The New Morris Method: an efficient second-order screening method , 2002, Reliab. Eng. Syst. Saf..

[16]  Berk Ustun,et al.  Importance Sampling in Stochastic Programming: A Markov Chain Monte Carlo Approach , 2015, INFORMS J. Comput..

[17]  Alexis Boukouvalas,et al.  An Efficient Screening Method for Computer Experiments , 2014, Technometrics.

[18]  Hong Wan,et al.  Simulation screening experiments using Lasso-optimal supersaturated design and analysis: A maritime operations application , 2013, 2013 Winter Simulations Conference (WSC).

[19]  Monica Menendez,et al.  Extending Morris method for qualitative global sensitivity analysis of models with dependent inputs , 2017, Reliab. Eng. Syst. Saf..

[20]  Susan M. Lewis,et al.  Design of experiments for screening , 2015, 1510.05248.

[21]  Max D. Morris,et al.  Factorial sampling plans for preliminary computational experiments , 1991 .

[22]  Michael Baron,et al.  Sequential Bonferroni Methods for Multiple Hypothesis Testing with Strong Control of Family-Wise Error Rates I and II , 2012 .

[23]  C. F. Jeff Wu,et al.  Experiments , 2021, Wiley Series in Probability and Statistics.

[24]  Barry L. Nelson,et al.  Improving the Efficiency and Efficacy of Controlled Sequential Bifurcation for Simulation Factor Screening , 2010, INFORMS J. Comput..

[25]  Gilles Pujol,et al.  Simplex-based screening designs for estimating metamodels , 2009, Reliab. Eng. Syst. Saf..

[26]  A. Wald.  Sequential Tests of Statistical Hypotheses , 1945 .

[27]  Joshua J. Millspaugh,et al.  Effects of sample size on kernel home range estimates , 1999 .

[28]  C. Antoniak,et al.  A distribution-free sequential probability-ratio test for multiple-resolution-element radars (Corresp.) , 1968, IEEE Trans. Inf. Theory.

[29]  M. Rendas,et al.  Extending Morris Method: identification of the interaction graph using cycle-equitabe designs , 2015 .

[30]  G. Deman,et al.  Sensitivity analysis of groundwater lifetime expectancy to hydro-dispersive parameters: The case of ANDRA Meuse/Haute-Marne site , 2015, Reliab. Eng. Syst. Saf..

[31]  J. Bartroff,et al.  Sequential Tests of Multiple Hypotheses Controlling Type I and II Familywise Error Rates. , 2013, Journal of statistical planning and inference.

[32]  M. Cuntz,et al.  Conditioning a Hydrologic Model Using Patterns of Remotely Sensed Land Surface Temperature , 2018 .

[33]  Roger David Braddock,et al.  The use of graph theory in the sensitivity analysis of the model output: a second order screening method , 1999 .

[34]  Zhaohong Deng,et al.  FRSDE: Fast reduced set density estimator using minimal enclosing ball approximation , 2008, Pattern Recognit..

[35]  Jack P. C. Kleijnen,et al.  State-of-the-Art Review: A User's Guide to the Brave New World of Designing Simulation Experiments , 2005, INFORMS J. Comput..

[36]  Bohyung Han,et al.  Sequential Kernel Density Approximation and Its Application to Real-Time Visual Tracking , 2008, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[37]  Theo Gasser,et al.  A Unifying Approach to Nonparametric Regression Estimation , 1988 .

[38]  Jack P. C. Kleijnen.  Design and Analysis of Simulation Experiments , 2007 .

[39]  R. Tibshirani,et al.  An introduction to the bootstrap , 1993 .

[40]  Michael Baron,et al.  Sequential tests controlling generalized familywise error rates , 2015 .

[41]  Andrea Saltelli,et al.  An effective screening design for sensitivity analysis of large models , 2007, Environ. Model. Softw..

[42]  Jack P. C. Kleijnen,et al.  Searching for important factors in simulation models with many factors: Sequential bifurcation , 1997 .

[43]  Danijel Skocaj,et al.  Online kernel density estimation for interactive learning , 2010, Image Vis. Comput..

[44]  Joachim Engel,et al.  The choice of weights in kernel regression estimation , 1990 .

[45]  Barry L. Nelson,et al.  Controlled Sequential Bifurcation: A New Factor-Screening Method for Discrete-Event Simulation , 2006, Oper. Res..

[46]  Zhenzhou Lu,et al.  A new effective screening design for structural sensitivity analysis of failure probability with the epistemic uncertainty , 2016, Reliab. Eng. Syst. Saf..

[47]  Thomas J. Santner,et al.  Two-Stage Sensitivity-Based Group Screening in Computer Experiments , 2012, Technometrics.

[48]  Benjamin Lamoureux,et al.  A combined sensitivity analysis and kriging surrogate modeling for early validation of health indicators , 2014, Reliab. Eng. Syst. Saf..

[49]  Susan M. Sanchez,et al.  So many factors, so little time...Simulation experiments in the frequency domain , 2006 .

[50]  Jack P. C. Kleijnen,et al.  Factor Screening for Simulation with Multiple Responses: Sequential Bifurcation , 2013, Eur. J. Oper. Res..

[51]  Kullback–Leibler Information , 2005 .

[52]  Chenggang Yu,et al.  A non-parametric sequential rank-sum probability ratio test method for binary hypothesis testing , 2004, Signal Process..