A review on advances in deep learning

Over the years conventional neural networks has shown state-of-art performance on many problems. However, their performance on recognition system is still not widely accepted in the machine learning community because these networks are unable to handle selectivity-invariance dilemma and also suffer from the problem of vanishing gradients. Some of these issues have been addressed by deep learning. Deep learning approaches attempt to disentangle intricate aspects of input by creating multiple levels of representation. These approaches have shown astonishing results in problem domains like recognition system, natural language processing, medical sciences, and in many other fields. The paper presents an overview of different deep learning approaches in a nutshell and also highlights some limitations which are restricting performance of deep neural networks in order to handle more realistic problems.

[1]  Geoffrey E. Hinton.  Where Do Features Come From? , 2014, Cogn. Sci..

[2]  Jonathan G. Fiscus,et al.  Darpa Timit Acoustic-Phonetic Continuous Speech Corpus CD-ROM {TIMIT} | NIST , 1993 .

[3]  Michael S. Bernstein,et al.  ImageNet Large Scale Visual Recognition Challenge , 2014, International Journal of Computer Vision.

[4]  Simon J. Doran,et al.  Stacked Autoencoders for Unsupervised Feature Learning and Multiple Organ Detection in a Pilot Study Using 4D Patient Data , 2013, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[5]  JEFFREY WOOD,et al.  Invariant pattern recognition: A review , 1996, Pattern Recognit..

[6]  Yoshua Bengio,et al.  Deep Sparse Rectifier Neural Networks , 2011, AISTATS.

[7]  Geoffrey E. Hinton,et al.  Reducing the Dimensionality of Data with Neural Networks , 2006, Science.

[8]  Joshua D. Lamos-Sweeney.  Deep learning using genetic algorithms , 2012 .

[9]  Pascal Vincent,et al.  Representation Learning: A Review and New Perspectives , 2012, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[10]  Yoshua Bengio,et al.  Knowledge Matters: Importance of Prior Information for Optimization , 2013, J. Mach. Learn. Res..

[11]  Tara N. Sainath,et al.  Improving deep neural networks for LVCSR using rectified linear units and dropout , 2013, 2013 IEEE International Conference on Acoustics, Speech and Signal Processing.

[12]  Antanas Verikas,et al.  Feature selection with neural networks , 2002, Pattern Recognit. Lett..

[13]  Guoqiang Peter Zhang,et al.  Neural networks for classification: a survey , 2000, IEEE Trans. Syst. Man Cybern. Part C.

[14]  Geoffrey E. Hinton.  Learning multiple layers of representation , 2007, Trends in Cognitive Sciences.

[15]  James Theiler,et al.  Online feature selection for pixel classification , 2005, ICML.

[16]  Gerald Penn,et al.  Convolutional Neural Networks for Speech Recognition , 2014, IEEE/ACM Transactions on Audio, Speech, and Language Processing.

[17]  Pietro Perona,et al.  Learning Generative Visual Models from Few Training Examples: An Incremental Bayesian Approach Tested on 101 Object Categories , 2004, 2004 Conference on Computer Vision and Pattern Recognition Workshop.

[18]  Michael J. Quinn,et al.  Parallel programming in C with MPI and OpenMP , 2003 .

[19]  Qiang Chen,et al.  Network In Network , 2013, ICLR.

[20]  Yoshua Bengio,et al.  Convolutional networks for images, speech, and time series , 1998 .

[21]  Huai Li,et al.  Artificial convolution neural network for medical image pattern recognition , 1995, Neural Networks.

[22]  Minyoung Kim,et al.  Deep Clustered Convolutional Kernels , 2015, FE@NIPS.

[23]  P. N. Suganthan,et al.  Differential Evolution: A Survey of the State-of-the-Art , 2011, IEEE Transactions on Evolutionary Computation.

[24]  Yoshua Bengio,et al.  Evolving Culture Versus Local Minima , 2014, Growing Adaptive Machines.

[25]  Yan Wu,et al.  A Simulation Study of Deep Belief Network Combined with the Self-Organizing Mechanism of Adaptive Resonance Theory , 2010, 2010 International Conference on Computational Intelligence and Software Engineering.

[26]  Andrzej Bargiela,et al.  Towards Evolutionary Deep Neural Networks , 2014, ECMS.

[27]  Li Fei-Fei,et al.  ImageNet: A large-scale hierarchical image database , 2009, CVPR.

[28]  Tom V. Mathew.  Genetic Algorithm , 2022 .

[29]  Tara N. Sainath,et al.  Optimization Techniques to Improve Training Speed of Deep Neural Networks for Large Speech Tasks , 2013, IEEE Transactions on Audio, Speech, and Language Processing.

[30]  Jürgen Schmidhuber,et al.  Multi-column deep neural networks for image classification , 2012, 2012 IEEE Conference on Computer Vision and Pattern Recognition.

[31]  Yoshua Bengio,et al.  Understanding the difficulty of training deep feedforward neural networks , 2010, AISTATS.

[32]  Quoc V. Le,et al.  On optimization methods for deep learning , 2011, ICML.

[33]  Yann LeCun,et al.  Learning Invariant Feature Hierarchies , 2012, ECCV Workshops.

[34]  Iddo Greental,et al.  Genetic algorithms for evolving deep neural networks , 2014, GECCO.

[35]  James Martens,et al.  Deep learning via Hessian-free optimization , 2010, ICML.

[36]  Yoshua Bengio,et al.  Random Search for Hyper-Parameter Optimization , 2012, J. Mach. Learn. Res..

[37]  Rob Fergus,et al.  Visualizing and Understanding Convolutional Networks , 2013, ECCV.

[38]  Geoffrey E. Hinton,et al.  ImageNet classification with deep convolutional neural networks , 2012, Commun. ACM.

[39]  Yoshua Bengio,et al.  Gradient-based learning applied to document recognition , 1998, Proc. IEEE.

[40]  Guigang Zhang,et al.  Deep Learning , 2016, Int. J. Semantic Comput..

[41]  Heekuck Oh,et al.  Neural Networks for Pattern Recognition , 1993, Adv. Comput..

[42]  Nitish Srivastava,et al.  Dropout: a simple way to prevent neural networks from overfitting , 2014, J. Mach. Learn. Res..

[43]  Alex Krizhevsky,et al.  Learning Multiple Layers of Features from Tiny Images , 2009 .

[44]  Kunihiko Fukushima,et al.  Neocognitron: A new algorithm for pattern recognition tolerant of deformations and shifts in position , 1982, Pattern Recognit..

[45]  Riccardo Poli,et al.  Particle swarm optimization , 1995, Swarm Intelligence.

[46]  Rong Jin,et al.  Online Feature Selection and Its Applications , 2014, IEEE Transactions on Knowledge and Data Engineering.

[47]  Wonyong Sung,et al.  Fixed point optimization of deep convolutional neural networks for object recognition , 2015, 2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[48]  Tijmen Tieleman,et al.  Training restricted Boltzmann machines using approximations to the likelihood gradient , 2008, ICML '08.

[49]  Brian Cheung,et al.  Hybrid Evolution of Convolutional Networks , 2011, 2011 10th International Conference on Machine Learning and Applications and Workshops.

[50]  Satish Kumar Jain,et al.  Neural networks : a classroom approach , 2005 .

[51]  Marco Dorigo,et al.  Ant colony optimization theory: A survey , 2005, Theor. Comput. Sci..

[52]  Lukasz A. Kurgan,et al.  Neural Networks in Bioinformatics , 2012, Handbook of Natural Computing.

[53]  Tara N. Sainath,et al.  Deep Convolutional Neural Networks for Large-scale Speech Tasks , 2015, Neural Networks.

[54]  Alex Krizhevsky,et al.  One weird trick for parallelizing convolutional neural networks , 2014, ArXiv.

[55]  T. Hampton,et al.  The Cancer Genome Atlas , 2020, Indian Journal of Medical and Paediatric Oncology.

[56]  Jürgen Schmidhuber,et al.  Deep learning in neural networks: An overview , 2014, Neural Networks.

[57]  John Shawe-Taylor,et al.  Representation Theory and Invariant Neural Networks , 1996, Discret. Appl. Math..

[58]  Yoshua Bengio,et al.  Deep Learning for NLP (without Magic) , 2012, ACL.

[59]  Huan Liu,et al.  Feature Selection for Classification , 1997, Intell. Data Anal..

[60]  Giovanni Montana,et al.  Deep neural networks for anatomical brain segmentation , 2015, 2015 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[61]  Andrew Zisserman,et al.  Very Deep Convolutional Networks for Large-Scale Image Recognition , 2014, ICLR.

[62]  D. Hubel,et al.  Receptive fields, binocular interaction and functional architecture in the cat's visual cortex , 1962, The Journal of physiology.

[63]  Goldberg,et al.  Genetic algorithms , 1993, Robust Control Systems with Genetic Algorithms.

[64]  Jack Dongarra,et al.  PVM: Parallel virtual machine: a users' guide and tutorial for networked parallel computing , 1995 .

[65]  Lawrence D. Jackel,et al.  Backpropagation Applied to Handwritten Zip Code Recognition , 1989, Neural Computation.

[66]  Marc'Aurelio Ranzato,et al.  Large Scale Distributed Deep Networks , 2012, NIPS.

[67]  Yann LeCun,et al.  The mnist database of handwritten digits , 2005 .

[68]  Geoffrey E. Hinton,et al.  Application of Deep Belief Networks for Natural Language Understanding , 2014, IEEE/ACM Transactions on Audio, Speech, and Language Processing.

[69]  Yaroslav Bulatov,et al.  Multi-digit Number Recognition from Street View Imagery using Deep Convolutional Neural Networks , 2013, ICLR.

[70]  Yee Whye Teh,et al.  A Fast Learning Algorithm for Deep Belief Nets , 2006, Neural Computation.

[71]  Anil K. Jain,et al.  Feature extraction methods for character recognition-A survey , 1996, Pattern Recognit..

[72]  Dumitru Erhan,et al.  Going deeper with convolutions , 2014, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[73]  Tara N. Sainath,et al.  FUNDAMENTAL TECHNOLOGIES IN MODERN SPEECH RECOGNITION Digital Object Identifier 10.1109/MSP.2012.2205597 , 2012 .

[74]  Manuel López-Ibáñez,et al.  Ant colony optimization , 2010, GECCO '10.

[75]  Miguel Torres,et al.  Feature Selection Using Artificial Neural Networks , 2008, MICAI.

[76]  Honglak Lee,et al.  Unsupervised learning of hierarchical representations with convolutional deep belief networks , 2011, Commun. ACM.

[77]  Yoshua Bengio,et al.  Gradient Flow in Recurrent Nets: the Difficulty of Learning Long-Term Dependencies , 2001 .