A Comparison of Factor Screening Methods for Simulation Models

YAESOUBI, REZA. A Comparison of Factor Screening Methods for Simulation Models. (Under the direction of Dr. Stephen D. Roberts). Computer simulation models that represent a real-world system consist of a large number of input variables which are generally referred to as factors in Design of Experiments (DOE). The large number of involved factors makes certain analyses which are usually conducted on the simulation models prohibitive or impractical. These analyses may include building predictive metamodels, finding the optimum factor settings for the simulated system, and etc. Factor Screening experiments are intended to examine all or some of the involved factors to identify those with significant effect on a selected response (output). The identified important factors can then be used in subsequent analyses. This thesis is focused on factor screening methods with promising performance on simulation models from the medical decision making community with a relatively large number of factors. Two groups of factor screening methods are addressed: classical designs which are generally used for physical systems, and recent designs which have been exclusively developed for simulation models. Among the classic designs, 2 Fractional Factorial (FF) Designs and Central Composite Designs are investigated in depth, because of their superior performance on the simulation models. Among the recent methods developed for simulation, Sequential Bifurcation (SB), folded-over SB (SB-X), Cheng’s method, Controlled Sequential Bifurcation (CSB), folded-over CSB (CSB-X), Latin Hypercube Designs (LHD), and Nearly Orthogonal Latin Hypercube (NOLH) designs are addressed. In addition, two methods based on Cheng’s method are developed in this thesis: the Modified Cheng’s method, and the folded-over Modified Cheng’s method (MCh-X). MCh-X is shown in this research that has superior performance compared with FF designs, Cheng’s method, and CSB-X for situations where the response has high homogeneous variance. Next, several criteria are considered for evaluating the factor screening methods, and the screening methods are compared based on the proposed criteria. Furthermore, the factor screening experiments are conducted on two available deterministic and stochastic simulation models. For the deterministic medical decision model, 2 Fractional Factorial Designs, folded-over SB (SB-X), and Nearly Orthogonal Latin Hypercube (NOLH) designs are used; and for the stochastic medical decision model, 2 Fractional Factorial Designs, folded-over Modified Cheng’s (MChe-X), and folded-over CSB (CSB-X) were applied Finally, based on quantitative measures, the performance of each method used for the available simulation models is evaluated in terms of its efficiency (requiring minimum number of runs), effectiveness (accuracy), and cost-effectiveness (achieving the highest accuracy with the least number of runs). Cost-effectiveness, which to the best of our knowledge has never been used as a criterion for evaluating factor screening methods, is introduced as a new measure encompassing both the concept of accuracy and efficiency. The research revealed that for the deterministic model, SB-X and for the stochastic model, MChX are the most cost-effective methods. A Comparison of Factor Screening Methods for Simulation Models