An Introduction to Multiobjective Simulation Optimization

The multiobjective simulation optimization (MOSO) problem is a nonlinear multiobjective optimization problem in which multiple simultaneous and conflicting objective functions can only be observed with stochastic error. We provide an introduction to MOSO at the advanced tutorial level, aimed at researchers and practitioners who wish to begin working in this emerging area. Our focus is exclusively on MOSO methods that characterize the entire efficient or Pareto-optimal set as the solution to the MOSO problem; later, this set may be used as input to the broader multicriteria decision-making process. Our introduction to MOSO includes an overview of existing theory, methods, and provably convergent algorithms that explicitly control sampling error for (1) MOSO on finite sets, called multiobjective ranking and selection; (2) MOSO with integer-ordered decision variables; and (3) MOSO with continuous decision variables. In the context of integer-ordered and continuous decision variables, we focus on methods that provably converge to a local efficient set under the natural ordering. We also discuss key open questions that remain in this emerging field.

[1]  Kalyanmoy Deb,et al.  Multi-objective Genetic Algorithms: Problem Difficulties and Construction of Test Problems , 1999, Evolutionary Computation.

[2]  Philippe Rigo,et al.  A review on simulation-based optimization methods applied to building performance analysis , 2014 .

[3]  Guy Feldman,et al.  Multi-objective ranking and selection: Optimal sampling laws and tractable approximations via SCORE , 2019, J. Simulation.

[4]  Hao Huang,et al.  Multiple objective probabilistic branch and bound for Pareto optimal approximation , 2014, Proceedings of the Winter Simulation Conference 2014.

[5]  Catherine M. Harmonosky,et al.  Optimization of systems with multiple performance measures via simulation: Survey and recommendations , 2008, Comput. Ind. Eng..

[6]  Loo Hay Lee,et al.  Stochastic Simulation Optimization - An Optimal Computing Budget Allocation , 2010, System Engineering and Operations Research.

[7]  Stacey L. Faulkenberg,et al.  On the quality of discrete representations in multiple objective programming , 2010 .

[8]  A. Messac,et al.  The normalized normal constraint method for generating the Pareto frontier , 2003 .

[9]  Walter J. Gutjahr A provably convergent heuristic for stochastic bicriteria integer programming , 2009, J. Heuristics.

[10]  Li-Ping Pang,et al.  Asymptotic convergence of stationary points of stochastic multiobjective programs with parametric variational inequality constraint via SAA approach , 2019, Journal of Industrial & Management Optimization.

[11]  Hamidreza Eskandari,et al.  Evolutionary multiobjective optimization in noisy problem environments , 2009, J. Heuristics.

[12]  Loo Hay Lee,et al.  A multi-objective genetic algorithm for robust flight scheduling using simulation , 2007, Eur. J. Oper. Res..

[13]  Eckart Zitzler,et al.  Evolutionary algorithms for multiobjective optimization: methods and applications , 1999 .

[14]  Kalyanmoy Deb,et al.  Hybrid Dynamic Resampling Algorithms for Evolutionary Multi-objective Optimization of Invariant-Noise Problems , 2016, EvoApplications.

[15]  Sujin Kim,et al.  The stochastic root-finding problem: Overview, solutions, and open questions , 2011, TOMC.

[16]  C.-C. Wang,et al.  Multi-objective simulation optimization for medical capacity allocation in emergency department , 2016, J. Simulation.

[17]  Kai Virtanen,et al.  Ranking and selection for multiple performance measures using incomplete preference information , 2015, Eur. J. Oper. Res..

[18]  Jürgen Branke,et al.  Evolutionary optimization in uncertain environments-a survey , 2005, IEEE Transactions on Evolutionary Computation.

[19]  Richard E. Rosenthal,et al.  Principles of multiobjective optimization , 1984 .

[20]  Edward J. Dudewicz,et al.  The heteroscedastic method: Multivariate implementation , 1987 .

[21]  Walter J. Gutjahr,et al.  Two Metaheuristics for Multiobjective Stochastic Combinatorial Optimization , 2005, SAGA.

[22]  Jörg Fliege,et al.  Robust multiobjective optimization & applications in portfolio optimization , 2014, Eur. J. Oper. Res..

[23]  Bernard Manderick,et al.  Exploration Versus Exploitation Trade-off in Infinite Horizon Pareto Multi-armed Bandits Algorithms , 2015, ICAART.

[24]  Yixin Chen,et al.  The Object-Oriented discrete event simulation modeling: A case study on aircraft spare part management , 2015, 2015 Winter Simulation Conference (WSC).

[25]  Matthias Ehrgott,et al.  Multicriteria Optimization , 2005 .

[26]  Loo Hay Lee,et al.  Multi-objective multi-fidelity optimization with ordinal transformation and optimal sampling , 2015, 2015 Winter Simulation Conference (WSC).

[27]  Lothar Thiele,et al.  Comparison of Multiobjective Evolutionary Algorithms: Empirical Results , 2000, Evolutionary Computation.

[28]  Raghu Pasupathy,et al.  Simulation Optimization: A Concise Overview and Implementation Guide , 2013 .

[29]  Jörg Fliege,et al.  Steepest descent methods for multicriteria optimization , 2000, Math. Methods Oper. Res..

[30]  Jie Xu,et al.  Industrial strength COMPASS: A comprehensive algorithm and software for optimization via simulation , 2010, TOMC.

[31]  Fouad Ben Abdelaziz,et al.  Solution approaches for the multiobjective stochastic programming , 2012, Eur. J. Oper. Res..

[32]  Gabriele Eichfelder,et al.  Adaptive Scalarization Methods in Multiobjective Optimization , 2008, Vector Optimization.

[33]  Jie Song,et al.  Integrating Optimal Simulation Budget Allocation and Genetic Algorithm to Find the Approximate Pareto Patient Flow Distribution , 2016, IEEE Transactions on Automation Science and Engineering.

[34]  M. Yoon,et al.  Single- and multi-objective ranking and selection procedures in simulation : a historical review , 2017 .

[35]  Susan R. Hunter,et al.  PyMOSO: Software for Multiobjective Simulation Optimization with R-PERLE and R-MinRLE , 2020, INFORMS J. Comput..

[36]  S. Jacobson,et al.  Techniques for simulation response optimization , 1989 .

[37]  R. Rosenthal Concepts, Theory, and Techniques PRINCIPLES OF MULTIOBJECTIVE OPTIMIZATION* , 1985 .

[38]  Peter W. Glynn,et al.  On Sampling Rates in Simulation-Based Recursions , 2018, SIAM J. Optim..

[39]  Gary B. Lamont,et al.  Evolutionary Algorithms for Solving Multi-Objective Problems , 2002, Genetic Algorithms and Evolutionary Computation.

[40]  Raghu Pasupathy,et al.  ASTRO-DF: A Class of Adaptive Sampling Trust-Region Algorithms for Derivative-Free Stochastic Optimization , 2016, SIAM J. Optim..

[41]  R. Pasupathy,et al.  ASTRO-DF: A CLASS OF ADAPTIVE SAMPLING TRUST-REGION ALGORITHMS FOR DERIVATIVE-FREE SIMULATION OPTIMIZATION , 2015 .

[42]  Margaret M. Wiecek,et al.  Covers and approximations in multiobjective optimization , 2017, J. Glob. Optim..

[43]  Linda Weiser Friedman,et al.  Design and analysis of multivariate response simulations: the state of the art , 1987 .

[44]  Honggang Wang,et al.  Zigzag Search for Continuous Multiobjective Optimization , 2013, INFORMS J. Comput..

[45]  Abhishek Singh,et al.  Uncertainty‐based multiobjective optimization of groundwater remediation design , 2003 .

[46]  Marcus Andersson,et al.  A web-based simulation optimization system for industrial scheduling , 2007, 2007 Winter Simulation Conference.

[47]  C. Anderson‐Cook,et al.  An Introduction to Multivariate Statistical Analysis (3rd ed.) (Book) , 2004 .

[48]  Bruce W. Schmeiser A practitioner, a vender, and a researcher walk into a bar: Trying to explain what researchers do , 2008, 2008 Winter Simulation Conference.

[49]  Gary B. Lamont,et al.  Evolutionary Algorithms for Solving Multi-Objective Problems (Genetic and Evolutionary Computation) , 2006 .

[50]  T. W. Anderson An Introduction to Multivariate Statistical Analysis , 1959 .

[51]  Kai Virtanen,et al.  Maintenance scheduling of a fleet of fighter aircraft through multi-objective simulation-optimization , 2014, Simul..

[52]  P. Serafini,et al.  Scalarizing vector optimization problems , 1984 .

[53]  Matthias Ehrgott,et al.  Multiple criteria decision analysis: state of the art surveys , 2005 .

[54]  Loo Hay Lee,et al.  Stochastically Constrained Ranking and Selection via SCORE , 2014, ACM Trans. Model. Comput. Simul..

[55]  Paul B. Reverdy,et al.  Gaussian multi-armed bandit problems with multiple objectives , 2016, 2016 American Control Conference (ACC).

[56]  Yang Tao,et al.  Multio-bjective ranking and selection based on hypervolume , 2016, 2016 Winter Simulation Conference (WSC).

[57]  J. Paris Performance assessment. , 1998, Journal of public health medicine.

[58]  Dimitri P. Bertsekas,et al.  Nonlinear Programming , 1997 .

[59]  M. Ehrgott Vilfredo Pareto and multi-objective optimization , 2012 .

[60]  Selçuk Gören,et al.  A Framework to Incorporate Decision-Maker Preferences Into Simulation Optimization to Support Collaborative Design , 2017, IEEE Transactions on Systems, Man, and Cybernetics: Systems.

[61]  Mustafa Y. Sir,et al.  Multi-objective simulation optimization using data envelopment analysis and genetic algorithm: Specific application to determining optimal resource levels in surgical services , 2013 .

[62]  Yaochu Jin,et al.  Surrogate-Assisted Multicriteria Optimization: Complexities, Prospective Solutions, and Business Case , 2017 .

[63]  Barry L. Nelson,et al.  Chapter 17 Selecting the Best System , 2006, Simulation.

[64]  Honggang Wang,et al.  Integer-Ordered Simulation Optimization using R-SPLINE: Retrospective Search with Piecewise-Linear Interpolation and Neighborhood Enumeration , 2013, TOMC.

[65]  Edward J. Dudewicz,et al.  The Heteroscedastic Method II: Simplified Sampling , 1991 .

[66]  Marco Laumanns,et al.  Scalable multi-objective optimization test problems , 2002, Proceedings of the 2002 Congress on Evolutionary Computation. CEC'02 (Cat. No.02TH8600).

[67]  Rong Jin,et al.  Stochastic Convex Optimization with Multiple Objectives , 2013, NIPS.

[68]  Charles Audet,et al.  Multiobjective Optimization Through a Series of Single-Objective Formulations , 2008, SIAM J. Optim..

[69]  Loo Hay Lee,et al.  Enhancement of supply chain resilience through inter-echelon information sharing , 2017 .

[70]  Weizhi Liu,et al.  A multi-objective perspective on robust ranking and selection , 2017, 2017 Winter Simulation Conference (WSC).

[71]  R. Pasupathy,et al.  A Guide to Sample Average Approximation , 2015 .

[72]  Nikolaos V. Sahinidis,et al.  Simulation optimization: a review of algorithms and applications , 2014, 4OR.

[73]  K. P. Sudheer,et al.  Adaptive multi-objective simulation–optimization framework for dynamic flood control operation in a river–reservoir system , 2015 .

[74]  C. Hwang Multiple Objective Decision Making - Methods and Applications: A State-of-the-Art Survey , 1979 .

[75]  José A. Sepúlveda,et al.  Multi-response simulation optimization using stochastic genetic search within a goal programming framework , 2000, 2000 Winter Simulation Conference Proceedings (Cat. No.00CH37165).

[76]  Hongwei Ding,et al.  A simulation-based multi-objective genetic algorithm approach for networked enterprises optimization , 2006, Eng. Appl. Artif. Intell..

[77]  Douglas J. Morrice,et al.  A Multiple Attribute Utility Theory Approach to Ranking and Selection , 2001, Manag. Sci..

[78]  Jörg Fliege,et al.  Newton's Method for Multiobjective Optimization , 2009, SIAM J. Optim..

[79]  Guy Feldman,et al.  Multi-objective simulation optimization on finite sets: Optimal allocation via scalarization , 2015, 2015 Winter Simulation Conference (WSC).

[80]  Alexander Shapiro,et al.  Lectures on Stochastic Programming: Modeling and Theory , 2009 .

[81]  Olivier Crespo,et al.  Original paper: Multiobjective optimization subject to uncertainty: Application to irrigation strategy management , 2010 .

[82]  K. Schittkowski,et al.  NONLINEAR PROGRAMMING , 2022 .

[83]  W. Stadler A survey of multicriteria optimization or the vector maximum problem, part I: 1776–1960 , 1979 .

[84]  Wenyu Wang Hong Wan,et al.  Sequential probability ratio test for Multiple-Objective Ranking and Selection , 2017, 2017 Winter Simulation Conference (WSC).

[85]  Loo Hay Lee,et al.  Optimal Computing Budget Allocation to Select the Nondominated Systems—A Large Deviations Perspective , 2018, IEEE Transactions on Automatic Control.

[86]  P. N. Rasmussen Tjalling C. Koopmans (edt.), Activity Analysis of Production and Allocation. Cowles Commission for Research in Economics, Monograph No. 13. John Wiley & Sons, New York, and Chapman & Hall, London, 1951. 404 sider. $ 4,50. , 1952 .

[87]  X. Gandibleux,et al.  Exact Methods for Multi-Objective Combinatorial Optimisation , 2016 .

[88]  Walter J. Gutjahr,et al.  Recent trends in metaheuristics for stochastic combinatorial optimization , 2011, Central European Journal of Computer Science.

[89]  Loo Hay Lee,et al.  Differentiated service inventory optimization using nested partitions and MOCBA , 2009, Comput. Oper. Res..

[90]  Layne T. Watson,et al.  Multiobjective optimization using an adaptive weighting scheme , 2016, Optim. Methods Softw..

[91]  Raghu Pasupathy,et al.  SimOpt: A library of simulation optimization problems , 2011, Proceedings of the 2011 Winter Simulation Conference (WSC).

[92]  Jorge Nocedal,et al.  Optimization Methods for Large-Scale Machine Learning , 2016, SIAM Rev..

[93]  Xin Yao,et al.  A benchmark test suite for evolutionary many-objective optimization , 2017, Complex & Intelligent Systems.

[94]  C. L. Philip Chen,et al.  I-Ching Divination Evolutionary Algorithm and its Convergence Analysis , 2017, IEEE Transactions on Cybernetics.

[95]  Amith Singhee,et al.  Pareto sampling: Choosing the right weights by derivative pursuit , 2010, Design Automation Conference.

[96]  F. Hank Grant,et al.  A Lexicographic Nelder-Mead simulation optimization method to solve multi-criteria problems , 2011, Comput. Ind. Eng..

[97]  R. H. Smith Optimization for Simulation : Theory vs . Practice , 2002 .

[98]  J. Fuhrer,et al.  Adapting agricultural land management to climate change: a regional multi-objective optimization approach , 2013, Landscape Ecology.

[99]  Bernard Manderick,et al.  Knowledge Gradient for Multi-objective Multi-armed Bandit Algorithms , 2014, ICAART.

[100]  Barry L. Nelson,et al.  Optimization via simulation over discrete decision variables , 2010 .

[101]  J. Dennis,et al.  A closer look at drawbacks of minimizing weighted sums of objectives for Pareto set generation in multicriteria optimization problems , 1997 .

[102]  Judy S. Lee Advances in simulation: validity and efficiency , 2014 .

[103]  Madalina M. Drugan,et al.  Correlated Gaussian Multi-Objective Multi-Armed Bandit Across Arms Algorithm , 2015, 2015 IEEE Symposium Series on Computational Intelligence.

[104]  Hao Huang,et al.  Discrete-event Simulation and Optimization to Improve the Performance of a Healthcare System , 2016 .

[105]  Jong-hyun Ryu,et al.  A Trust-Region Algorithm for Bi-Objective Stochastic Optimization , 2011, ICCS.

[106]  Jeffrey A. Joines,et al.  Supply chain multi-objective simulation optimization , 2002, Proceedings of the Winter Simulation Conference.

[107]  Jean-Antoine Désidéri,et al.  Descent algorithm for nonsmooth stochastic multiobjective optimization , 2017, Comput. Optim. Appl..

[108]  J. Désidéri Multiple-gradient descent algorithm (MGDA) for multiobjective optimization , 2012 .

[109]  Barry L. Nelson,et al.  A brief introduction to optimization via simulation , 2009, Proceedings of the 2009 Winter Simulation Conference (WSC).

[110]  Güzin Bayraksan,et al.  Stochastic Constraints and Variance Reduction Techniques , 2015 .

[111]  Gerald W. Evans,et al.  Multicriteria optimization of simulation models , 1991, 1991 Winter Simulation Conference Proceedings..

[112]  Mahmoud H. Alrefaei,et al.  A simulated annealing technique for multi-objective simulation optimization , 2009, Appl. Math. Comput..

[113]  Henri Bonnel,et al.  Stochastic Optimization over a Pareto Set Associated with a Stochastic Multi-Objective Optimization Problem , 2014, J. Optim. Theory Appl..

[114]  Barry L. Nelson,et al.  Discrete Optimization via Simulation Using COMPASS , 2006, Oper. Res..

[115]  Jason R. W. Merrick,et al.  Using multiattribute utility theory to avoid bad outcomes by focusing on the best systems in ranking and selection , 2015, J. Simulation.

[116]  Catherine M. Harmonosky,et al.  A simulation optimization method that considers uncertainty and multiple performance measures , 2007, Eur. J. Oper. Res..

[117]  Matthias Ehrgott,et al.  Computation of ideal and Nadir values and implications for their use in MCDM methods , 2003, Eur. J. Oper. Res..

[118]  Barry L. Nelson,et al.  Using Ranking and Selection to "Clean Up" after Simulation Optimization , 2003, Oper. Res..

[119]  Susan R. Hunter,et al.  Biobjective Simulation Optimization on Integer Lattices Using the Epsilon-Constraint Method in a Retrospective Approximation Framework , 2020, INFORMS J. Comput..

[120]  Honggang Wang,et al.  Multi-objective retrospective optimization using stochastic zigzag search , 2017, Eur. J. Oper. Res..

[121]  Bernard Manderick,et al.  The scalarized multi-objective multi-armed bandit problem: An empirical study of its exploration vs. exploitation tradeoff , 2014, 2014 International Joint Conference on Neural Networks (IJCNN).

[122]  Susan R. Hunter,et al.  Parallel Ranking and Selection , 2017 .

[123]  Fangfang Sun,et al.  A comparison of two metamodel-based methodologies for multiple criteria simulation optimization using an injection molding case study , 2013 .

[124]  Marco Laumanns,et al.  Performance assessment of multiobjective optimizers: an analysis and review , 2003, IEEE Trans. Evol. Comput..

[125]  Loo Hay Lee,et al.  Integration of indifference-zone with multi-objective computing budget allocation , 2010, Eur. J. Oper. Res..

[126]  Kalyanmoy Deb,et al.  Solving Bilevel Multi-Objective Optimization Problems Using Evolutionary Algorithms , 2009, EMO.

[127]  D. J. White,et al.  A Bibliography on the Applications of Mathematical Programming Multiple-objective Methods , 1990 .

[128]  A. Shapiro Monte Carlo Sampling Methods , 2003 .

[129]  Chun-Hung Chen,et al.  Simulation Budget Allocation for Further Enhancing the Efficiency of Ordinal Optimization , 2000, Discret. Event Dyn. Syst..

[130]  María Guadalupe Villarreal Marroquín,et al.  A multicriteria simulation optimization method for injection molding , 2011, Proceedings of the 2011 Winter Simulation Conference (WSC).

[131]  Chun-Hung Chen,et al.  Stochastic Simulation Optimization for Discrete Event Systems:Perturbation Analysis, Ordinal Optimization, and Beyond , 2013 .

[132]  Chenhao Zhou,et al.  A simulation-based vessel-truck coordination strategy for lighterage terminals , 2018, Transportation Research Part C: Emerging Technologies.

[133]  David Goldsman A Practical Guide to Ranking and Selection Methods , 2015 .

[134]  Jürgen Branke,et al.  A new myopic sequential sampling algorithm for multi-objective problems , 2015, 2015 Winter Simulation Conference (WSC).

[135]  Warren B. Powell,et al.  Simulation optimization: A panel on the state of the art in research and practice , 2014, Proceedings of the Winter Simulation Conference 2014.

[136]  Susan R. Hunter,et al.  Maximizing quantitative traits in the mating design problem via simulation-based Pareto estimation , 2016 .

[137]  P. Koumoutsakos,et al.  Multiobjective evolutionary algorithm for the optimization of noisy combustion processes , 2002 .

[138]  Lionel Amodeo,et al.  Comparison of Metaheuristic Approaches for Multi-objective Simulation-Based Optimization in Supply Chain Inventory Management , 2009, EvoWorkshops.

[139]  José A. Sepúlveda,et al.  Multi-objective simulation optimization for a cancer treatment center , 2001, Proceeding of the 2001 Winter Simulation Conference (Cat. No.01CH37304).

[140]  Johannes Jahn,et al.  Vector optimization - theory, applications, and extensions , 2004 .

[141]  Barry L. Nelson,et al.  Discrete Optimization via Simulation , 2015 .

[142]  Ching-Lai Hwang,et al.  Methods for Multiple Objective Decision Making , 1979 .

[143]  Jong-hyun Ryu,et al.  The sample average approximation method for multi-objective stochastic optimization , 2011, Proceedings of the 2011 Winter Simulation Conference (WSC).

[144]  Marco Laumanns,et al.  Scalable Test Problems for Evolutionary Multiobjective Optimization , 2005, Evolutionary Multiobjective Optimization.

[145]  Loo Hay Lee,et al.  A multi-objective selection procedure of determining a Pareto set , 2009, Comput. Oper. Res..

[146]  Kaisa Miettinen,et al.  Nonlinear multiobjective optimization , 1998, International series in operations research and management science.

[147]  H. Kushner,et al.  Stochastic Approximation and Recursive Algorithms and Applications , 2003 .

[148]  Ihsan Sabuncuoglu,et al.  Simulation optimization: A comprehensive review on theory and applications , 2004 .

[149]  L. Lee,et al.  MO-COMPASS: a fast convergent search algorithm for multi-objective discrete optimization via simulation , 2015 .

[150]  R. Pasupathy,et al.  The Adaptive Sampling Gradient Method Optimizing Smooth Functions with an Inexact Oracle , 2017 .

[151]  Peter I. Frazier,et al.  Guessing preferences: A new approach to multi-attribute ranking and selection , 2011, Proceedings of the 2011 Winter Simulation Conference (WSC).

[152]  Qingfu Zhang,et al.  Multiobjective evolutionary algorithms: A survey of the state of the art , 2011, Swarm Evol. Comput..

[153]  Edward J. Dudewicz,et al.  Multivariate ranking and selection without reduction to a univariate problem , 1978, WSC '78.

[154]  Jie Zhang,et al.  Consistencies and Contradictions of Performance Metrics in Multiobjective Optimization , 2014, IEEE Transactions on Cybernetics.

[155]  Guy Feldman Sampling Laws for Multi-Objective Simulation Optimization on Finite Sets , 2017 .

[156]  F. Fred Choobineh,et al.  Mean-variance based ranking and selection , 2010, Proceedings of the 2010 Winter Simulation Conference.

[157]  R. K. Ursem Multi-objective Optimization using Evolutionary Algorithms , 2009 .

[158]  Stefano Lucidi,et al.  A Simulation-Based Multiobjective Optimization Approach for Health Care Service Management , 2016, IEEE Transactions on Automation Science and Engineering.

[159]  Lee Loo Hay,et al.  Application of Evolutionary Algorithms for Solving Multi-Objective Simulation Optimization Problems , 2009 .

[160]  Luca Maria Gambardella,et al.  A survey on metaheuristics for stochastic combinatorial optimization , 2009, Natural Computing.

[161]  Carlos M. Fonseca,et al.  A Box Decomposition Algorithm to Compute the Hypervolume Indicator , 2015, Comput. Oper. Res..

[162]  Panos Y. Papalambros,et al.  A multi-objective optimization framework for assessing military ground vehicle design for safety , 2014 .

[163]  Serpil Sayin,et al.  Computing the nadir point for multiobjective discrete optimization problems , 2015, J. Glob. Optim..

[164]  Mustafa Akan,et al.  A Broader View of Designing the Liver Allocation System , 2012, Oper. Res..

[165]  Michael C. Fu,et al.  Optimization via simulation: A review , 1994, Ann. Oper. Res..

[166]  Martin Kozek,et al.  Simulation-based multi-objective system optimization of train traction systems , 2017, Simul. Model. Pract. Theory.

[167]  R. Lyndon While,et al.  A review of multiobjective test problems and a scalable test problem toolkit , 2006, IEEE Transactions on Evolutionary Computation.

[168]  Averill M. Law,et al.  Simulation Modeling and Analysis , 1982 .

[169]  Sigrún Andradóttir,et al.  A Review of Random Search Methods , 2015 .

[170]  Guy Feldman,et al.  SCORE Allocations for Bi-objective Ranking and Selection , 2018, ACM Trans. Model. Comput. Simul..

[171]  Hong Zhang Multi-objective simulation-optimization for earthmoving operations , 2008 .

[172]  Edward J. Dudewicz,et al.  A multivariate solution of the multivariate ranking and selection problem , 1981 .

[173]  Jong-hyun Ryu,et al.  A Derivative-Free Trust-Region Method for Biobjective Optimization , 2014, SIAM J. Optim..

[174]  Dimitris Bertsimas,et al.  Fairness, Efficiency, and Flexibility in Organ Allocation for Kidney Transplantation , 2013, Oper. Res..

[175]  Jörg Fliege,et al.  Stochastic Multiobjective Optimization: Sample Average Approximation and Applications , 2011, J. Optim. Theory Appl..

[176]  John E. Dennis,et al.  Normal-Boundary Intersection: A New Method for Generating the Pareto Surface in Nonlinear Multicriteria Optimization Problems , 1998, SIAM J. Optim..

[177]  Loo Hay Lee,et al.  Multi-objective simulation-based evolutionary algorithm for an aircraft spare parts allocation problem , 2008, Eur. J. Oper. Res..

[178]  Loo Hay Lee,et al.  Multi-objective optimization for a hospital inpatient flow process via Discrete Event Simulation , 2015, 2015 Winter Simulation Conference (WSC).

[179]  Lijun Liu,et al.  Estimation of the Pareto front in stochastic simulation through stochastic Kriging , 2017, Simul. Model. Pract. Theory.

[180]  Peter Kall,et al.  Stochastic Programming , 1995 .

[181]  Subhash Suri,et al.  On Klee's measure problem for grounded boxes , 2012, SoCG '12.

[182]  Bernard Manderick,et al.  Thompson Sampling in the Adaptive Linear Scalarized Multi Objective Multi Armed Bandit , 2015, ICAART.

[183]  Kaisa Miettinen,et al.  A solution process for simulation-based multiobjective design optimization with an application in the paper industry , 2014, Comput. Aided Des..

[184]  Loo Hay Lee,et al.  Simulation Optimization: A Review and Exploration in the New Era of Cloud Computing and Big Data , 2015, Asia Pac. J. Oper. Res..

[185]  Tag Gon Kim,et al.  Pareto Set Selection for Multiobjective Stochastic Simulation Model , 2020, IEEE Transactions on Systems, Man, and Cybernetics: Systems.

[186]  Serpil Sayin,et al.  Measuring the quality of discrete representations of efficient sets in multiple objective mathematical programming , 2000, Math. Program..

[187]  Dimo Brockhoff,et al.  Theoretical Aspects of Evolutionary Multiobjective Optimization , 2011, Theory of Randomized Search Heuristics.

[188]  Raghu Pasupathy,et al.  A Testbed of Simulation-Optimization Problems , 2006, Proceedings of the 2006 Winter Simulation Conference.

[189]  Michael C. Fu,et al.  Handbook of Simulation Optimization , 2014 .

[190]  L. Lee,et al.  Finding the non-dominated Pareto set for multi-objective simulation models , 2010 .

[191]  Deepak Gupta,et al.  Maufacturing supply chain applications 1: supply chain multi-objective simulation optimization , 2002, WSC '02.

[192]  Jürgen Branke,et al.  Sequential Sampling to Myopically Maximize the Expected Value of Information , 2010, INFORMS J. Comput..

[193]  Panos M. Pardalos,et al.  A survey of recent developments in multiobjective optimization , 2007, Ann. Oper. Res..

[194]  Alexander Shapiro,et al.  Lectures on Stochastic Programming - Modeling and Theory, Second Edition , 2014, MOS-SIAM Series on Optimization.

[195]  Jasbir S. Arora,et al.  Survey of multi-objective optimization methods for engineering , 2004 .

[196]  Matthias Ehrgott,et al.  Continuous Multiobjective Programming , 2016 .

[197]  W. David Kelton,et al.  Sequential selection procedures: Using sample means to improve efficiency , 2005, Eur. J. Oper. Res..

[198]  Urmila M. Diwekar,et al.  Stochastic modeling and multi-objective optimization for the APECS system , 2011, Comput. Chem. Eng..

[199]  Walter J. Gutjahr,et al.  Stochastic multi-objective optimization: a survey on non-scalarizing methods , 2016, Ann. Oper. Res..

[200]  A. Ruszczynski Stochastic Programming Models , 2003 .

[201]  Yacov Y. Haimes,et al.  Multiobjective Decision Making: Theory and Methodology , 1983 .

[202]  Raghu Pasupathy,et al.  Root finding via DARTS — Dynamic Adaptive Random Target Shooting , 2010, Proceedings of the 2010 Winter Simulation Conference.

[203]  Jean Bigeon,et al.  Performance indicators in multiobjective optimization , 2018, Eur. J. Oper. Res..

[204]  Sigurdur Olafsson,et al.  Chapter 21 Metaheuristics , 2006, Simulation.

[205]  Henri Bonnel,et al.  Optimization over the Pareto outcome set associated with a convex bi-objective optimization problem: theoretical results, deterministic algorithm and application to the stochastic case , 2015, J. Glob. Optim..

[206]  Shane G. Henderson,et al.  An introduction to simulation optimization , 2015, 2015 Winter Simulation Conference (WSC).

[207]  Timothy M. Chan Klee's Measure Problem Made Easy , 2013, 2013 IEEE 54th Annual Symposium on Foundations of Computer Science.

[208]  Manuel Chica,et al.  Why Simheuristics? Benefits, Limitations, and Best Practices When Combining Metaheuristics with Simulation , 2017, SSRN Electronic Journal.

[209]  Susan R. Hunter,et al.  An epsilon-constraint method for integer-ordered bi-objective simulation optimization , 2017, 2017 Winter Simulation Conference (WSC).

[210]  Michael C. Fu,et al.  Feature Article: Optimization for simulation: Theory vs. Practice , 2002, INFORMS J. Comput..

[211]  W.-H. Feng,et al.  A simulation study of cadaveric liver allocation with a single-score patient prioritization formula , 2013, J. Simulation.

[212]  Shapour Azarm,et al.  Metrics for Quality Assessment of a Multiobjective Design Optimization Solution Set , 2001 .