Nature Inspired Meta-heuristic Algorithms for Deep Learning: Recent Progress and Novel Perspective

Deep learning is presently attracting extra ordinary attention from both the industry and the academia. The application of deep learning in computer vision has recently gain popularity. The optimization of deep learning models through nature inspired algorithms is a subject of debate in computer science. The application areas of the hybrid of natured inspired algorithms and deep learning architecture includes: machine vision and learning, image processing, data science, autonomous vehicles, medical image analysis, biometrics, etc. In this paper, we present recent progress on the application of nature inspired algorithms in deep learning. The survey pointed out recent development issues, strengths, weaknesses and prospects for future research. A new taxonomy is created based on natured inspired algorithms for deep learning. The trend of the publications in this domain is depicted; it shows the research area is growing but slowly. The deep learning architectures not exploit by the nature inspired algorithms for optimization are unveiled. We believed that the survey can facilitate synergy between the nature inspired algorithms and deep learning research communities. As such, massive attention can be expected in a near future.

[1]  Sameem Abdul Kareem,et al.  MALARIA SEVERITY CLASSIFICATION THROUGH JORDAN-ELMAN NEURAL NETWORK BASED ON FEATURES EXTRACTED FROM THICK BLOOD SMEAR , 2015 .

[2]  Alejandro Baldominos Gómez,et al.  Evolutionary convolutional neural networks: An application to handwriting recognition , 2017, Neurocomputing.

[3]  Siddharth Swarup Rautaray,et al.  Application of Deep Learning for Object Detection , 2018 .

[4]  Jacob Russell Neterer,et al.  Deep Learning in Natural Language Processing , 2018, Proceedings of the West Virginia Academy of Science.

[5]  Ivor W. Tsang,et al.  Learning word dependencies in text by means of a deep recurrent belief network , 2016, Knowl. Based Syst..

[6]  Xin-She Yang,et al.  Learning Parameters in Deep Belief Networks Through Firefly Algorithm , 2016, ANNPR.

[7]  Pascal Vincent,et al.  Representation Learning: A Review and New Perspectives , 2012, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[8]  Dervis Karaboga,et al.  Artificial Bee Colony (ABC) Optimization Algorithm for Training Feed-Forward Neural Networks , 2007, MDAI.

[9]  Hak-Keung Lam,et al.  Tuning of the structure and parameters of a neural network using an improved genetic algorithm , 2003, IEEE Trans. Neural Networks.

[10]  Haizhou Li,et al.  A Cost-Sensitive Deep Belief Network for Imbalanced Classification , 2018, IEEE Transactions on Neural Networks and Learning Systems.

[11]  Hicham Chaoui,et al.  State of Charge and State of Health Estimation for Lithium Batteries Using Recurrent Neural Networks , 2017, IEEE Transactions on Vehicular Technology.

[12]  Chia-Feng Juang,et al.  A hybrid of genetic algorithm and particle swarm optimization for recurrent network design , 2004, IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics).

[13]  Junfei Qiao,et al.  A self-organizing deep belief network for nonlinear system modeling , 2018, Appl. Soft Comput..

[14]  Chuang Sun,et al.  Discriminative Deep Belief Networks with Ant Colony Optimization for Health Status Assessment of Machine , 2017, IEEE Transactions on Instrumentation and Measurement.

[15]  Simon Fong,et al.  How meta-heuristic algorithms contribute to deep learning in the hype of big data analytics , 2018 .

[16]  José García Rodríguez,et al.  A survey on deep learning techniques for image and video semantic segmentation , 2018, Appl. Soft Comput..

[17]  Linzhen Nie,et al.  Longitudinal speed control of autonomous vehicle based on a self-adaptive PID of radial basis function neural network , 2018 .

[18]  Petr Dolezel,et al.  Pattern recognition neural network as a tool for pest birds detection , 2016, 2016 IEEE Symposium Series on Computational Intelligence (SSCI).

[19]  Yoshua Bengio,et al.  Generative Adversarial Nets , 2014, NIPS.

[20]  Bruce Denby,et al.  Updating the silent speech challenge benchmark with deep learning , 2017, Speech Commun..

[21]  Ping Wang,et al.  Research on Deep Learning Techniques in Breaking Text-Based Captchas and Designing Image-Based Captcha , 2018, IEEE Transactions on Information Forensics and Security.

[22]  Geoffrey E. Hinton,et al.  Reducing the Dimensionality of Data with Neural Networks , 2006, Science.

[23]  Xin-She Yang,et al.  Fine-tuning deep belief networks using cuckoo search , 2016 .

[24]  Mohamad Ivan Fanany,et al.  Simulated Annealing Algorithm for Deep Learning , 2015 .

[25]  Xin-She Yang,et al.  Cuckoo Search via Lévy flights , 2009, 2009 World Congress on Nature & Biologically Inspired Computing (NaBIC).

[26]  Yuxin Zhao,et al.  Swarm Intelligence: Today and Tomorrow , 2016, 2016 3rd International Conference on Soft Computing & Machine Intelligence (ISCMI).

[27]  W. Pitts,et al.  A Logical Calculus of the Ideas Immanent in Nervous Activity (1943) , 2021, Ideas That Created the Future.

[28]  Kai Liu,et al.  Deep Boltzmann Machines Aided Design Based on Genetic Algorithms , 2014 .

[29]  Jürgen Schmidhuber,et al.  Deep learning in neural networks: An overview , 2014, Neural Networks.

[30]  Xin-She Yang,et al.  Quaternion-based Deep Belief Networks fine-tuning , 2017, Appl. Soft Comput..

[31]  Haruna Chiroma,et al.  Neural networks optimization through genetic algorithm searches: A review , 2017 .

[32]  Rabab Kreidieh Ward,et al.  Deep learning for pixel-level image fusion: Recent advances and future prospects , 2018, Inf. Fusion.

[33]  Kunikazu Kobayashi,et al.  Time Series Forecasting Using Restricted Boltzmann Machine , 2012, ICIC.

[34]  John H. Holland,et al.  Adaptation in Natural and Artificial Systems: An Introductory Analysis with Applications to Biology, Control, and Artificial Intelligence , 1992 .

[35]  Geoffrey E. Hinton,et al.  Deep Learning , 2015, Nature.

[36]  Kasiprasad Mannepalli,et al.  A novel Adaptive Fractional Deep Belief Networks for speaker emotion recognition , 2017 .

[37]  Yee Whye Teh,et al.  A Fast Learning Algorithm for Deep Belief Nets , 2006, Neural Computation.

[38]  João Paulo Papa,et al.  On the Model Selection of Bernoulli Restricted Boltzmann Machines Through Harmony Search , 2015, GECCO.

[39]  João Paulo Papa,et al.  Model selection for Discriminative Restricted Boltzmann Machines through meta-heuristic techniques , 2015, J. Comput. Sci..

[40]  Ashiq Anjum,et al.  Deep Learning Hyper-Parameter Optimization for Video Analytics in Clouds , 2019, IEEE Transactions on Systems, Man, and Cybernetics: Systems.

[41]  Bo Xing,et al.  Innovative Computational Intelligence: A Rough Guide to 134 Clever Algorithms , 2013 .

[42]  Nathan S. Netanyahu,et al.  Genetic algorithms and deep learning for automatic painter classification , 2014, GECCO.

[43]  Arash Bahrammirzaee,et al.  A comparative survey of artificial intelligence applications in finance: artificial neural networks, expert system and hybrid intelligent systems , 2010, Neural Computing and Applications.

[44]  Lei Wang,et al.  Ensemble One-Dimensional Convolution Neural Networks for Skeleton-Based Action Recognition , 2018, IEEE Signal Processing Letters.

[45]  Radu-Emil Precup,et al.  Embedding Gravitational Search Algorithms in Convolutional Neural Networks for OCR applications , 2012, 2012 7th IEEE International Symposium on Applied Computational Intelligence and Informatics (SACI).

[46]  Nazri Mohd Nawi,et al.  A New Back-Propagation Neural Network Optimized with Cuckoo Search Algorithm , 2013, ICCSA.