Generation of Optimal Artificial Neural Networks Using a Pattern Search Algorithm: Application to Approximation of Chemical Systems

A pattern search optimization method is applied to the generation of optimal artificial neural networks (ANNs). Optimization is performed using a mixed variable extension to the generalized pattern search method. This method offers the advantage that categorical variables, such as neural transfer functions and nodal connectivities, can be used as parameters in optimization. When used together with a surrogate, the resulting algorithm is highly efficient for expensive objective functions. Results demonstrate the effectiveness of this method in optimizing an ANN for the number of neurons, the type of transfer function, and the connectivity among neurons. The optimization method is applied to a chemistry approximation of practical relevance. In this application, temperature and a chemical source term are approximated as functions of two independent parameters using optimal ANNs. Comparison of the performance of optimal ANNs with conventional tabulation methods demonstrates equivalent accuracy by considerable savings in memory storage. The architecture of the optimal ANN for the approximation of the chemical source term consists of a fully connected feedforward network having four nonlinear hidden layers and 117 synaptic weights. An equivalent representation of the chemical source term using tabulation techniques would require a 500 500 grid point discretization of the parameter space.

[1]  M. Abramson Mixed Variable Optimization of a Load-Bearing Thermal Insulation System Using a Filter Pattern Search Algorithm , 2004 .

[2]  Norberto Fueyo,et al.  An economical strategy for storage of chemical kinetics: Fitting in situ adaptive tabulation with artificial neural networks , 2000 .

[3]  Søren Nymand Lophaven,et al.  DACE - A Matlab Kriging Toolbox, Version 2.0 , 2002 .

[4]  Stephen B. Pope,et al.  Computationally efficient implementation of combustion chemistry using in situ adaptive tabulation , 1997 .

[5]  Martin T. Hagan,et al.  Neural network design , 1995 .

[6]  Laurene V. Fausett,et al.  Fundamentals Of Neural Networks , 1994 .

[7]  Charles Audet,et al.  A Pattern Search Filter Method for Nonlinear Programming without Derivatives , 2001, SIAM J. Optim..

[8]  Assaad R. Masri,et al.  Artificial neural network implementation of chemistry with pdf simulation of H2/CO2 flames , 1996 .

[9]  Bernhard Sendhoff,et al.  Structure optimization of neural networks for evolutionary design optimization , 2005, Soft Comput..

[10]  Stephen B. Pope,et al.  An integrated PDF/neural network approach for simulating turbulent reacting systems , 1996 .

[11]  Ulrich Maas,et al.  Simplifying chemical kinetics: Intrinsic low-dimensional manifolds in composition space , 1992 .

[12]  Peter M. Todd,et al.  Designing Neural Networks using Genetic Algorithms , 1989, ICGA.

[13]  Stefan Bornholdt,et al.  General asymmetric neural networks and structure design by genetic algorithms: a learning rule for temporal patterns , 1992, Proceedings of IEEE Systems Man and Cybernetics Conference - SMC.

[14]  A. J. Booker,et al.  A rigorous framework for optimization of expensive functions by surrogates , 1998 .

[15]  Marcus Frean,et al.  The Upstart Algorithm: A Method for Constructing and Training Feedforward Neural Networks , 1990, Neural Computation.

[16]  J.-Y. Chen,et al.  A self-organizing-map approach to chemistry representation in combustion applications , 2000 .

[17]  Johannes Janicka,et al.  LES using artificial neural networks for chemistry representation , 2005 .

[18]  Richard J. Beckman,et al.  A Comparison of Three Methods for Selecting Values of Input Variables in the Analysis of Output From a Computer Code , 2000, Technometrics.

[19]  Dirk Gorissen,et al.  Multiobjective global surrogate modeling, dealing with the 5-percent problem , 2010, Engineering with Computers.

[20]  Michael Frenklach,et al.  PRISM: piecewise reusable implementation of solution mapping. An economical strategy for chemical kinetics , 1998 .

[21]  John R. Koza,et al.  Genetic generation of both the weights and architecture for a neural network , 1991, IJCNN-91-Seattle International Joint Conference on Neural Networks.

[22]  Sam Kwong,et al.  Genetic structure for NN topology and weights optimization , 1995 .

[23]  Simon Haykin,et al.  Neural Networks: A Comprehensive Foundation , 1998 .

[24]  Charles Audet,et al.  Generalized pattern searches with derivative information , 2002, Math. Program..

[25]  Jaehong Park,et al.  Evolutionary projection neural networks , 1997, Proceedings of 1997 IEEE International Conference on Evolutionary Computation (ICEC '97).

[26]  S. H. Lam,et al.  Understanding complex chemical kinetics with computational singular perturbation , 1989 .

[27]  Virginia Torczon,et al.  On the Convergence of Pattern Search Algorithms , 1997, SIAM J. Optim..

[28]  Xin Yao,et al.  Evolutionary design of artificial neural networks with different nodes , 1996, Proceedings of IEEE International Conference on Evolutionary Computation.

[29]  Norberto Fueyo,et al.  Modelling the Temporal Evolution of a Reduced Combustion Chemical System With an Artificial Neural Network , 1998 .

[30]  Xin Yao,et al.  Evolving artificial neural networks , 1999, Proc. IEEE.

[31]  Norberto Fueyo,et al.  A single-step time-integrator of a methane-air chemical system using artificial neural networks , 1999 .

[32]  Yi-jian Liu,et al.  Modeling of hydraulic turbine systems based on a Bayesian-Gaussian neural network driven by sliding window data , 2009, Journal of Zhejiang University SCIENCE C.

[33]  Meng Wang,et al.  Optimal Aeroacoustic Shape Design Using the Surrogate Management Framework , 2003 .

[34]  J. Dennis,et al.  Mixed Variable Optimization of the Number and Composition of Heat Intercepts in a Thermal Insulation System , 2001 .

[35]  Charles Audet,et al.  Pattern Search Algorithms for Mixed Variable Programming , 2000, SIAM J. Optim..

[36]  N. Peters Laminar diffusion flamelet models in non-premixed turbulent combustion , 1984 .

[37]  Peter J. Angeline,et al.  An evolutionary algorithm that constructs recurrent neural networks , 1994, IEEE Trans. Neural Networks.

[38]  John E. Dennis,et al.  A framework for managing models in nonlinear optimization of computationally expensive functions , 1999 .

[39]  M. D. McKay,et al.  A comparison of three methods for selecting values of input variables in the analysis of output from a computer code , 2000 .

[40]  Christoph Schmitt,et al.  Optimal artificial neural networks and tabulation methods for chemistry representation in LES of a bluff-body swirl-stabilized flame , 2009 .

[41]  Marco Sciandrone,et al.  An Algorithm Model for Mixed Variable Programming , 2005, SIAM J. Optim..

[42]  Art B. Owen,et al.  9 Computer experiments , 1996, Design and analysis of experiments.

[43]  Michael C. Mozer,et al.  Skeletonization: A Technique for Trimming the Fat from a Network via Relevance Assessment , 1988, NIPS.

[44]  Charles Audet,et al.  Analysis of Generalized Pattern Searches , 2000, SIAM J. Optim..

[45]  J. Hammersley SIMULATION AND THE MONTE CARLO METHOD , 1982 .