An improved dynamic structure-based neural networks determination approaches to simulation optimization problems

Simulation optimization studies the problem of optimizing simulation-based objectives. This field has a strong history in engineering but often suffers from several difficulties including being time-consuming and NP-hardness. Simulation optimization is a new and hot topic in the field of system simulation and operational research. This paper presents a hybrid approach that combines Evolutionary Algorithms with neural networks (NNs) for solving simulation optimization problems. In this hybrid approach, we use NNs to replace the known simulation model for evaluating subsequent iterative solutions. Further, we apply the dynamic structure-based neural networks to learn and replace the known simulation model. The determination of dynamic structure-based neural networks is the kernel of this paper. The final experimental results demonstrated that the proposed approach can find optimal or close-to-optimal solutions and is superior to other recent algorithms in simulation optimization.

[1]  Fred W. Glover,et al.  Simulation optimization: a review, new developments, and applications , 2005, Proceedings of the Winter Simulation Conference, 2005..

[2]  Jack P. C. Kleijnen,et al.  Simulation and optimization in production planning: A case study , 1993, Decis. Support Syst..

[3]  Kyung K. Choi,et al.  A new response surface methodology for reliability-based design optimization , 2004 .

[4]  Douglas C. Montgomery,et al.  Optimizing protocol interaction using response surface methodology , 2006, IEEE Transactions on Mobile Computing.

[5]  F. Hunt,et al.  Sample path optimality for a Markov optimization problem , 2005 .

[6]  Stephen M. Robinson,et al.  Sample-path optimization of convex stochastic performance functions , 1996, Math. Program..

[7]  John Moody,et al.  Architecture Selection Strategies for Neural Networks: Application to Corporate Bond Rating Predicti , 1995, NIPS 1995.

[8]  Won-Sun Ruy,et al.  Polynomial genetic programming for response surface modeling Part 1: a methodology , 2005 .

[9]  Sigrún Andradóttir,et al.  Chapter 20 An Overview of Simulation Optimization via Random Search , 2006, Simulation.

[10]  Bo K. Wong,et al.  Neural network applications in finance: A review and analysis of literature (1990-1996) , 1998, Inf. Manag..

[11]  Stephen E. Chick,et al.  New Two-Stage and Sequential Procedures for Selecting the Best Simulated System , 2001, Oper. Res..

[12]  Michael C. Fu,et al.  Convergence of simultaneous perturbation stochastic approximation for nondifferentiable optimization , 2003, IEEE Trans. Autom. Control..

[13]  Stephen M. Robinson,et al.  Analysis of Sample-Path Optimization , 1996, Math. Oper. Res..

[14]  Yim-Shu Lee,et al.  A genetic algorithm based variable structure Neural Network , 2003, IECON'03. 29th Annual Conference of the IEEE Industrial Electronics Society (IEEE Cat. No.03CH37468).

[15]  Jing Liu,et al.  A multiagent genetic algorithm for global numerical optimization , 2004, IEEE Trans. Syst. Man Cybern. Part B.

[16]  T. R. Neelakantan,et al.  NEURAL NETWORK-BASED SIMULATION-OPTIMIZATION MODEL FOR RESERVOIR OPERATION , 2000 .

[17]  Pandu R. Tadikamalla,et al.  Output maximization of a CIM system: simulation and statistical approach , 1993 .

[18]  Nabil Derbel,et al.  Variable structure neural networks for adaptive control of nonlinear systems using the stochastic approximation , 2006, Simul. Model. Pract. Theory.

[19]  Michalis E. Zervakis,et al.  High-order neural network structure selection for function approximation applications using genetic algorithms , 2004, IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics).

[20]  Dirk P. Kroese,et al.  The Cross-Entropy Method: A Unified Approach to Combinatorial Optimization, Monte-Carlo Simulation and Machine Learning , 2004 .

[21]  Youshen Xia,et al.  A recurrent neural network for nonlinear convex optimization subject to nonlinear inequality constraints , 2004, IEEE Trans. Circuits Syst. I Regul. Pap..

[22]  Ihsan Sabuncuoglu,et al.  Simulation optimization: A comprehensive review on theory and applications , 2004 .

[23]  F.H.F. Leung,et al.  Tuning of the structure and parameters of neural network using an improved genetic algorithm , 2001, IECON'01. 27th Annual Conference of the IEEE Industrial Electronics Society (Cat. No.37243).

[24]  Giuseppe D'Acquisto,et al.  Cross-entropy-based adaptive optimization of simulation parameters for Markovian-driven service systems , 2005, Simul. Model. Pract. Theory.

[25]  Seong-Hee Kim,et al.  Performance of variance updating ranking and selection procedures , 2005, Proceedings of the Winter Simulation Conference, 2005..

[26]  Alexander Shapiro,et al.  Solving multistage asset investment problems by the sample average approximation method , 2006, Math. Program..

[27]  Irène Charon,et al.  The noising methods: A generalization of some metaheuristics , 2001, Eur. J. Oper. Res..

[28]  Wangchao Li,et al.  A variable structure neural network model and its applications , 1993, Proceedings of TENCON '93. IEEE Region 10 International Conference on Computers, Communications and Automation.

[29]  Christos G. Cassandras,et al.  Ordinal optimisation and simulation , 2000, J. Oper. Res. Soc..

[30]  Miguel A. Mariño,et al.  Ant colony optimization algorithm (ACO): a new heuristic approach for engineering optimization , 2005 .

[31]  Marko Kegl,et al.  An efficient gradient‐based optimization algorithm for mechanical systems , 2002 .

[32]  Cheng-Yan Kao,et al.  An evolutionary algorithm for large traveling salesman problems , 2004, IEEE Trans. Syst. Man Cybern. Part B.

[33]  Peter Greistorfer,et al.  A Tabu Scatter Search Metaheuristic for the Arc Routing Problem , 2002 .

[34]  Hadar I. Avi-Itzhak,et al.  High Accuracy Optical Character Recognition Using Neural Networks with Centroid Dithering , 1995, IEEE Trans. Pattern Anal. Mach. Intell..

[35]  Tung-Kuan Liu,et al.  Tuning the structure and parameters of a neural network by using hybrid Taguchi-genetic algorithm , 2006, IEEE Trans. Neural Networks.

[36]  J. A. Lozano,et al.  Estimation of Distribution Algorithms: A New Tool for Evolutionary Computation , 2001 .

[37]  Yu-Chi Ho,et al.  Ordinal optimization of DEDS , 1992, Discret. Event Dyn. Syst..

[38]  Henri Pierreval,et al.  Distributed evolutionary algorithms for simulation optimization , 2000, IEEE Trans. Syst. Man Cybern. Part A.

[39]  H. Akaike A new look at the statistical model identification , 1974 .

[40]  Dale E. Seborg,et al.  Model structure determination in neural network models , 2000 .

[41]  Stephen E. Chick,et al.  New Procedures to Select the Best Simulated System Using Common Random Numbers , 2001, Manag. Sci..

[42]  Behrokh Khoshnevis,et al.  Development of a Rapid Prototyping System using Response Surface Methodology , 2006, Qual. Reliab. Eng. Int..

[43]  Talal M. Alkhamis,et al.  Simulation-based optimization using simulated annealing with ranking and selection , 2002, Comput. Oper. Res..

[44]  Tung-Kuan Liu,et al.  Hybrid Taguchi-genetic algorithm for global numerical optimization , 2004, IEEE Transactions on Evolutionary Computation.

[45]  Geoffrey E. Hinton,et al.  An Alternative Model for Mixtures of Experts , 1994, NIPS.

[46]  A. Lindenmayer Mathematical models for cellular interactions in development. II. Simple and branching filaments with two-sided inputs. , 1968, Journal of theoretical biology.

[47]  Serge Domenech,et al.  Use of genetic algorithms and gradient based optimization techniques for calcium phosphate precipitation , 2004 .

[48]  Lee W. Schruben,et al.  A survey of recent advances in discrete input parameter discrete-event simulation optimization , 2004 .

[49]  Sailing He,et al.  A gradient‐based optimization approach to the inverse problem for multi‐layered structures , 1999 .

[50]  S.D. Hill,et al.  Simulation optimization of airline delay using simultaneous perturbation stochastic approximation , 2000, Proceedings 33rd Annual Simulation Symposium (SS 2000).

[51]  Peter Kemper,et al.  Combining Response Surface Methodology with Numerical Methods for Optimization of Markovian Models , 2006, IEEE Transactions on Dependable and Secure Computing.

[52]  William Y. Svrcek,et al.  Automatic design of neural network structures , 2001 .

[53]  Kenneth Runesson,et al.  Parameter estimation for a viscoplastic damage model using a gradient‐based optimization algorithm , 1998 .

[54]  Jacques Teghem,et al.  Using metaheuristics for solving a production scheduling problem in a chemical firm. A case study , 1996 .

[55]  Leyuan Shi,et al.  Nested Partitions Method for Global Optimization , 2000, Oper. Res..

[56]  Enver Yücesan,et al.  Discrete-event simulation optimization using ranking, selection, and multiple comparison procedures: A survey , 2003, TOMC.

[57]  Jean B. Lasserre Sample-path average optimality for Markov control processes , 1999, IEEE Trans. Autom. Control..

[58]  Shalabh Bhatnagar,et al.  Adaptive multivariate three-timescale stochastic approximation algorithms for simulation based optimization , 2005, TOMC.

[59]  J. Spall,et al.  Optimal random perturbations for stochastic approximation using a simultaneous perturbation gradient approximation , 1997, Proceedings of the 1997 American Control Conference (Cat. No.97CH36041).

[60]  Barry L. Nelson,et al.  Using Ranking and Selection to "Clean Up" after Simulation Optimization , 2003, Oper. Res..

[61]  Douglas J. Morrice,et al.  A Multiple Attribute Utility Theory Approach to Ranking and Selection , 2001, Manag. Sci..

[62]  Siegfried Selberherr,et al.  An extensible TCAD optimization framework combining gradient based and genetic optimizers , 2002 .

[63]  Sigurdur Ólafsson,et al.  Iterative ranking-and-selection for large-scale optimization , 1999, WSC '99.

[64]  Yixin Chen,et al.  Hybrid constrained simulated annealing and genetic algorithms for nonlinear constrained optimization , 2001, Proceedings of the 2001 Congress on Evolutionary Computation (IEEE Cat. No.01TH8546).

[65]  G. Jivotovski A gradient based heuristic algorithm and its application to discrete optimization of bar structures , 2000 .

[66]  Alexander Shapiro,et al.  The Sample Average Approximation Method Applied to Stochastic Routing Problems: A Computational Study , 2003, Comput. Optim. Appl..

[67]  R. Ravi,et al.  Hedging Uncertainty: Approximation Algorithms for Stochastic Optimization Problems , 2004, Math. Program..

[68]  Meir J. Rosenblatt,et al.  A COMBINED OPTIMIZATION AND SIMULATION APPROACH FOR DESIGNING AUTOMATED STORAGE/RETRIEVAL SYSTEMS , 1993 .

[69]  R. Dwight,et al.  Effect of Approximations of the Discrete Adjoint on Gradient-Based Optimization , 2006 .

[70]  Toshihide Ibaraki,et al.  Metaheuristics as robust and simple optimization tools , 1996, Proceedings of IEEE International Conference on Evolutionary Computation.

[71]  Jesús Cid-Sueiro,et al.  A model selection algorithm for a posteriori probability estimation with neural networks , 2005, IEEE Transactions on Neural Networks.

[72]  André I. Khuri,et al.  Response surface methodology , 2010 .

[73]  Lucy Y. Pao,et al.  Computing budget allocation for efficient ranking and selection of variances with application to target tracking algorithms , 2004, IEEE Transactions on Automatic Control.

[74]  Zhaoheng Liu,et al.  Optimization of mechanical systems reliability using ant colony based simulation approach , 2005 .

[75]  Shie Mannor,et al.  A Tutorial on the Cross-Entropy Method , 2005, Ann. Oper. Res..

[76]  Connie M. Borror,et al.  Response Surface Methodology: A Retrospective and Literature Survey , 2004 .

[77]  Shalabh Bhatnagar,et al.  A Discrete Parameter Stochastic Approximation Algorithm for Simulation Optimization , 2005, Simul..

[78]  Thomas Stützle,et al.  Ant Colony Optimization , 2009, EMO.

[79]  Yong Lu,et al.  A robust stochastic genetic algorithm (StGA) for global numerical optimization , 2004, IEEE Transactions on Evolutionary Computation.

[80]  R. H. Myers,et al.  Response Surface Methodology: Process and Product Optimization Using Designed Experiments , 1995 .

[81]  M. Marchesi,et al.  Tabu search metaheuristics for electromagnetic problems optimization in continuous domains , 1999 .

[82]  Stephen M. Rock,et al.  Gradient‐based parameter optimization for systems containing discrete‐valued functions , 2002 .

[83]  J. Spall,et al.  Optimal random perturbations for stochastic approximation using a simultaneous perturbation gradient approximation , 1997, Proceedings of the 1997 American Control Conference (Cat. No.97CH36041).

[84]  Yianni Attikiouzel,et al.  A novel multicriteria optimization algorithm for the structure determination of multilayer feedforward neural networks , 1996 .

[85]  Hongwei Ding,et al.  A simulation optimization methodology for supplier selection problem , 2005, Int. J. Comput. Integr. Manuf..

[86]  N.V. Bhat,et al.  Modeling chemical process systems via neural computation , 1990, IEEE Control Systems Magazine.

[87]  A. Lindenmayer Mathematical models for cellular interactions in development. I. Filaments with one-sided inputs. , 1968, Journal of theoretical biology.

[88]  Denis José Schiozer,et al.  Scatter Search Metaheuristic Applied to the History Matching Problem , 2006 .

[89]  J. Rissanen A UNIVERSAL PRIOR FOR INTEGERS AND ESTIMATION BY MINIMUM DESCRIPTION LENGTH , 1983 .

[90]  P. Sánchez,et al.  PROCEEDINGS OF THE 2003 WINTER SIMULATION CONFERENCE , 2016 .

[91]  E. Polak,et al.  Reliability-based optimal design using sample average approximations , 2004 .

[92]  Liqun Qi,et al.  A Gradient-based Continuous Method for Large-scale Optimization Problems , 2005, J. Glob. Optim..

[93]  Fan-wen,et al.  Exponential convergence of sample average approximation methods for a class of stochastic mathematical programs with complementarity constraints , 2006 .

[94]  Guang-Bin Huang,et al.  Neuron selection for RBF neural network classifier based on data structure preserving criterion , 2005, IEEE Transactions on Neural Networks.

[95]  Gang George Yin,et al.  Regime Switching Stochastic Approximation Algorithms with Application to Adaptive Discrete Stochastic Optimization , 2004, SIAM J. Optim..

[96]  Jiaqiao Hu,et al.  A Model Reference Adaptive Search Algorithm for Global Optimization , 2005 .

[97]  Jing Wei,et al.  Sample average approximation methods for stochastic MINLPs , 2004, Comput. Chem. Eng..

[98]  K. Preston White,et al.  Stochastic approximation with simulated annealing as an approach to global discrete-event simulation optimization , 2004, Proceedings of the 2004 Winter Simulation Conference, 2004..

[99]  J. Spall,et al.  Simulation-Based Optimization with Stochastic Approximation Using Common Random Numbers , 1999 .

[100]  Da-Nian Luan,et al.  Diagnosis of abrupt faults using variable‐structure neural network , 2000 .

[101]  M. Bakr,et al.  Accelerated gradient-based optimization of planar circuits , 2005, IEEE Transactions on Antennas and Propagation.

[102]  Bo K. Wong,et al.  Neural network applications in business: A review and analysis of the literature (1988-1995) , 1997, Decis. Support Syst..

[103]  El-Ghazali Talbi,et al.  A Taxonomy of Hybrid Metaheuristics , 2002, J. Heuristics.

[104]  Youshen Xia,et al.  An Extended Projection Neural Network for Constrained Optimization , 2004, Neural Computation.

[105]  Klaus-Robert Müller,et al.  Asymptotic statistical theory of overtraining and cross-validation , 1997, IEEE Trans. Neural Networks.

[106]  Krzysztof Fleszar,et al.  An evolutionary algorithm for resource-constrained project scheduling , 2002, IEEE Trans. Evol. Comput..

[107]  Amos H. C. Ng,et al.  Simulation-based optimisation using local search and neural network metamodels , 2006, Artificial Intelligence and Soft Computing.

[108]  Michael C. Fu,et al.  Stochastic optimization using model reference adaptive search , 2005, Proceedings of the Winter Simulation Conference, 2005..

[109]  Jack P. C. Kleijnen,et al.  Kriging for interpolation in random simulation , 2003, J. Oper. Res. Soc..

[110]  Ying-Tung Hsiao,et al.  A novel dynamic structural neural network with neuro-regeneration and neuro-degeneration , 2005, 2005 9th International Workshop on Cellular Neural Networks and Their Applications.

[111]  J. Ford,et al.  Hybrid estimation of distribution algorithm for global optimization , 2004 .

[112]  Mansooreh Mollaghasemi,et al.  A genetic algorithm and an indifference-zone ranking and selection framework for simulation optimization , 2001, Proceeding of the 2001 Winter Simulation Conference (Cat. No.01CH37304).

[113]  H. Kitano Neurogenetic learning: an integrated method of designing and training neural networks using genetic algorithms , 1994 .

[114]  Ling Wang,et al.  A hybrid genetic algorithm-neural network strategy for simulation optimization , 2005, Appl. Math. Comput..

[115]  Wee Ser,et al.  Probabilistic neural-network structure determination for pattern classification , 2000, IEEE Trans. Neural Networks Learn. Syst..

[116]  R. Brennan,et al.  Stochastic optimization applied to a manufacturing system operation problem , 1995, Winter Simulation Conference Proceedings, 1995..

[117]  Jürgen Branke,et al.  New developments in ranking and selection: an empirical comparison of the three main approaches , 2005, Proceedings of the Winter Simulation Conference, 2005..

[118]  Junsheng Liang,et al.  Study on simulation-based optimization for flip chip package parameters by using RSM analysis and Ant algorithm , 2003, Fifth International Conference onElectronic Packaging Technology Proceedings, 2003. ICEPT2003..

[119]  Aristid Lindenmayer,et al.  Mathematical Models for Cellular Interactions in Development , 1968 .

[120]  Todd A. Sriver,et al.  Combined pattern search and ranking and selection for simulation optimization , 2004, Proceedings of the 2004 Winter Simulation Conference, 2004..

[121]  Barry L. Nelson,et al.  Discrete Optimization via Simulation Using COMPASS , 2006, Oper. Res..

[122]  P. A. Simionescu,et al.  Constrained optimization problem solving using estimation of distribution algorithms , 2004, Proceedings of the 2004 Congress on Evolutionary Computation (IEEE Cat. No.04TH8753).

[123]  Belle R. Upadhyaya,et al.  Sensor monitoring using a fuzzy neural network with an automatic structure constructor , 2003 .

[124]  Toshio Fukuda,et al.  Theory and applications of neural networks for industrial control systems , 1992, IEEE Trans. Ind. Electron..

[125]  Yuping Wang,et al.  An orthogonal genetic algorithm with quantization for global numerical optimization , 2001, IEEE Trans. Evol. Comput..