Automated Neural Network Construction with Similarity Sensitive Evolutionary Algorithms

Deep learning has been successfully applied to a wide variety of tasks. It generates reusable knowledge that allows transfer learning to significantly impact more scientific research areas. However, there is no automatic way to build a new model that guarantees an adequate performance. In this paper, we propose an automated neural network construction framework to overcome the limitations found in current approaches using transfer learning. Currently, researchers spend much time and effort to understand the characteristics of the data when designing a new network model. Therefore, the proposed method leverages the strength in evolutionary algorithms to automate the search and optimization process. Similarities between the individuals are also considered during the cycled evolutionary process to avoid sticking to a local optimal. Overall, the experimental results effectively reach optimal solutions proving that a time-consuming task could also be done by an automated process that exceeds the human ability to select the best hyperparameters.

[1]  Thomas Fang Zheng,et al.  Transfer learning for speech and language processing , 2015, 2015 Asia-Pacific Signal and Information Processing Association Annual Summit and Conference (APSIPA).

[2]  Qiang Yang,et al.  A Survey on Transfer Learning , 2010, IEEE Transactions on Knowledge and Data Engineering.

[3]  Frank Hutter,et al.  CMA-ES for Hyperparameter Optimization of Deep Neural Networks , 2016, ArXiv.

[4]  F.H.F. Leung,et al.  Tuning of the structure and parameters of neural network using an improved genetic algorithm , 2001, IECON'01. 27th Annual Conference of the IEEE Industrial Electronics Society (Cat. No.37243).

[5]  Elliot Meyerson,et al.  Evolving Deep Neural Networks , 2017, Artificial Intelligence in the Age of Neural Networks and Brain Computing.

[6]  Ramesh Raskar,et al.  Designing Neural Network Architectures using Reinforcement Learning , 2016, ICLR.

[7]  Matthew J. Hausknecht,et al.  Beyond short snippets: Deep networks for video classification , 2015, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[8]  Kenneth O. Stanley,et al.  Deep Neuroevolution: Genetic Algorithms Are a Competitive Alternative for Training Deep Neural Networks for Reinforcement Learning , 2017, ArXiv.

[9]  Alan L. Yuille,et al.  Genetic CNN , 2017, 2017 IEEE International Conference on Computer Vision (ICCV).

[10]  S. Sitharama Iyengar,et al.  Automatic Convolutional Neural Network Selection for Image Classification Using Genetic Algorithms , 2018, 2018 IEEE International Conference on Information Reuse and Integration (IRI).

[11]  Risto Miikkulainen,et al.  Evolving Neural Networks through Augmenting Topologies , 2002, Evolutionary Computation.

[12]  Quoc V. Le,et al.  Neural Architecture Search with Reinforcement Learning , 2016, ICLR.

[13]  Michael I. Jordan,et al.  Deep Transfer Learning with Joint Adaptation Networks , 2016, ICML.

[14]  Erik Cambria,et al.  Recent Trends in Deep Learning Based Natural Language Processing , 2017, IEEE Comput. Intell. Mag..

[15]  Masanori Suganuma,et al.  A genetic programming approach to designing convolutional neural network architectures , 2017, GECCO.

[16]  Yoshua Bengio,et al.  Random Search for Hyper-Parameter Optimization , 2012, J. Mach. Learn. Res..

[17]  Tara N. Sainath,et al.  Deep Neural Networks for Acoustic Modeling in Speech Recognition , 2012 .