Recurrent neural network ensembles for convergence prediction in surrogate-assisted evolutionary optimization

Evaluating the fitness of candidate solutions in evolutionary algorithms can be computationally expensive when the fitness is determined using an iterative numerical process. This paper illustrates how an ensemble of Recurrent Neural Networks can be used as a robust surrogate to predict converged Computational Fluid Dynamics data from unconverged data. The training of the individual neural networks is controlled and a variance range is used to determine if the surrogates have been adequately trained to predict diverse and accurate solutions. Heterogeneous ensemble members are used due to the limited data available and results show that for certain parameters, predictions can be made to within 5% of the converged data's final output, using approximately 40% of the iterations needed for convergence. The implications of the method and results presented are that it is possible to use ensembles of Recurrent Neural Networks to provide accurate fitness predictions for an evolutionary algorithm and that they could be used to reduce the time needed to achieve optimal designs based on time-consuming Computational Fluid Dynamics simulations.

[1]  Xin Yao,et al.  DIVACE: Diverse and Accurate Ensemble Learning Algorithm , 2004, IDEAL.

[2]  John Fulcher,et al.  Computational Intelligence: An Introduction , 2008, Computational Intelligence: A Compendium.

[3]  Bernhard Sendhoff,et al.  Reducing Fitness Evaluations Using Clustering Techniques and Neural Network Ensembles , 2004, GECCO.

[4]  Oleksandr Makeyev,et al.  Neural network with ensembles , 2010, The 2010 International Joint Conference on Neural Networks (IJCNN).

[5]  Bernhard Sendhoff,et al.  Generalizing Surrogate-Assisted Evolutionary Computation , 2010, IEEE Transactions on Evolutionary Computation.

[6]  Bernhard Sendhoff,et al.  A framework for evolutionary optimization with approximate fitness functions , 2002, IEEE Trans. Evol. Comput..

[7]  Bernhard Sendhoff,et al.  Neural network regularization and ensembling using multi-objective evolutionary algorithms , 2004, Proceedings of the 2004 Congress on Evolutionary Computation (IEEE Cat. No.04TH8753).

[8]  David W. Opitz,et al.  Actively Searching for an E(cid:11)ective Neural-Network Ensemble , 1996 .

[9]  Yaochu Jin,et al.  Heterogeneous classifier ensembles for EEG-based motor imaginary detection , 2012, 2012 12th UK Workshop on Computational Intelligence (UKCI).

[10]  Yaochu Jin,et al.  Surrogate-assisted evolutionary computation: Recent advances and future challenges , 2011, Swarm Evol. Comput..

[11]  Bernhard Sendhoff,et al.  Prediction of convergence dynamics of design performance using differential recurrent neural networks , 2008, 2008 IEEE International Joint Conference on Neural Networks (IEEE World Congress on Computational Intelligence).

[12]  R. Haftka,et al.  Ensemble of surrogates , 2007 .

[13]  Simon Haykin,et al.  Neural Networks and Learning Machines , 2010 .

[14]  Andy J. Keane,et al.  Optimization using surrogate models and partially converged computational fluid dynamics simulations , 2006, Proceedings of the Royal Society A: Mathematical, Physical and Engineering Sciences.

[15]  Xin Yao,et al.  Evolving artificial neural network ensembles , 2008, IEEE Computational Intelligence Magazine.

[16]  Wei Tang,et al.  Ensembling neural networks: Many could be better than all , 2002, Artif. Intell..

[17]  K. Stewartson.  Mechanics of fluids , 1978, Nature.

[18]  Peter Stagge,et al.  Recurrent neural networks for time series classification , 2003, Neurocomputing.

[19]  Weeratunge Malalasekera,et al.  An introduction to computational fluid dynamics - the finite volume method , 2007 .

[20]  Yaochu Jin,et al.  A comprehensive survey of fitness approximation in evolutionary computation , 2005, Soft Comput..

[21]  Leslie S. Smith,et al.  A novel neural network ensemble architecture for time series forecasting , 2011, Neurocomputing.

[22]  Nathan Intrator,et al.  Optimal ensemble averaging of neural networks , 1997 .

[23]  Antony Browne,et al.  Neural network ensembles: combining multiple models for enhanced performance using a multistage approach , 2004, Expert Syst. J. Knowl. Eng..

[24]  Barbara Hammer,et al.  Neural Smithing – Supervised Learning in Feedforward Artificial Neural Networks , 2001, Pattern Analysis & Applications.

[25]  Néstor V. Queipo,et al.  Toward an optimal ensemble of kernel-based approximations with engineering applications , 2006, The 2006 IEEE International Joint Conference on Neural Network Proceedings.

[26]  Piet Demeester,et al.  A Surrogate Modeling and Adaptive Sampling Toolbox for Computer Based Design , 2010, J. Mach. Learn. Res..

[27]  Bernhard Sendhoff,et al.  A systems approach to evolutionary multiobjective structural optimization and beyond , 2009, IEEE Computational Intelligence Magazine.

[28]  Xin Yao,et al.  Diversity creation methods: a survey and categorisation , 2004, Inf. Fusion.

[29]  Yuan Tian,et al.  Modeling and optimal control of a batch polymerization reactor using a hybrid stacked recurrent neural network model , 2001 .