Deep Machine Learning - A New Frontier in Artificial Intelligence Research [Research Frontier]

This article provides an overview of the mainstream deep learning approaches and research directions proposed over the past decade. It is important to emphasize that each approach has strengths and "weaknesses, depending on the application and context in "which it is being used. Thus, this article presents a summary on the current state of the deep machine learning field and some perspective into how it may evolve. Convolutional Neural Networks (CNNs) and Deep Belief Networks (DBNs) (and their respective variations) are focused on primarily because they are well established in the deep learning field and show great promise for future work.

[1]  Kunihiko Fukushima,et al.  Neocognitron: A self-organizing neural network model for a mechanism of pattern recognition unaffected by shift in position , 1980, Biological Cybernetics.

[2]  李幼升,et al.  Ph , 1989 .

[3]  K. De Jong,et al.  Evolving intelligent agents: A 50 year quest , 2008, IEEE Computational Intelligence Magazine.

[4]  Yann LeCun,et al.  Synergistic Face Detection and Pose Estimation with Energy-Based Models , 2004, J. Mach. Learn. Res..

[5]  Dileep George,et al.  How the brain might work: a hierarchical and temporal model for learning and recognition , 2008 .

[6]  Abdesselam Bouzerdoum,et al.  A new class of convolutional neural networks (SICoNNets) and their application of face detection , 2003, Proceedings of the International Joint Conference on Neural Networks, 2003..

[7]  Jeffrey W. Miller,et al.  Biomimetic sensory abstraction using hierarchical quilted self-organizing maps , 2006, SPIE Optics East.

[8]  Hossein Mobahi,et al.  Deep Learning via Semi-supervised Embedding , 2012, Neural Networks: Tricks of the Trade.

[9]  Yoshua Bengio,et al.  Extracting and composing robust features with denoising autoencoders , 2008, ICML '08.

[10]  Geoffrey E. Hinton,et al.  Phoneme recognition using time-delay neural networks , 1989, IEEE Trans. Acoust. Speech Signal Process..

[11]  E. Rolls,et al.  INVARIANT FACE AND OBJECT RECOGNITION IN THE VISUAL SYSTEM , 1997, Progress in Neurobiology.

[12]  Honglak Lee,et al.  Convolutional deep belief networks for scalable unsupervised learning of hierarchical representations , 2009, ICML '09.

[13]  R. Weale.  Vision. A Computational Investigation Into the Human Representation and Processing of Visual Information. David Marr , 1983 .

[14]  J. Hawkins,et al.  On Intelligence , 2004 .

[15]  Sven Behnke,et al.  Hierarchical Neural Networks for Image Interpretation (Lecture Notes in Computer Science) , 2003 .

[16]  D. Hubel,et al.  Receptive fields, binocular interaction and functional architecture in the cat's visual cortex , 1962, The Journal of physiology.

[17]  Patrice Y. Simard,et al.  Best practices for convolutional neural networks applied to visual document analysis , 2003, Seventh International Conference on Document Analysis and Recognition, 2003. Proceedings..

[18]  T. Poggio,et al.  Hierarchical models of object recognition in cortex , 1999, Nature Neuroscience.

[19]  Marc'Aurelio Ranzato,et al.  Unsupervised Learning of Invariant Feature Hierarchies with Applications to Object Recognition , 2007, 2007 IEEE Conference on Computer Vision and Pattern Recognition.

[20]  Geoffrey E. Hinton,et al.  Learning Multilevel Distributed Representations for High-Dimensional Sequences , 2007, AISTATS.

[21]  Alan J Lockett and Risto Miikkulainen.  Temporal Convolution Machines for Sequence Learning , 2009 .

[22]  D. Mumford,et al.  The role of the primary visual cortex in higher level vision , 1998, Vision Research.

[23]  Sean R Eddy,et al.  What is dynamic programming? , 2004, Nature Biotechnology.

[24]  Yoshua Bengio,et al.  Gradient-based learning applied to document recognition , 1998, Proc. IEEE.

[25]  Andy Adler,et al.  Comparing Human and Automatic Face Recognition Performance , 2007, IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics).

[26]  K. A. De Jong,et al.  Evolving intelligent agents: A 50 year quest , 2008, IEEE Comput. Intell. Mag..

[27]  Geoffrey E. Hinton,et al.  Reducing the Dimensionality of Data with Neural Networks , 2006, Science.

[28]  Geoffrey E. Hinton.  Training Products of Experts by Minimizing Contrastive Divergence , 2002, Neural Computation.

[29]  Kunihiko Fukushima,et al.  Neocognitron for handwritten digit recognition , 2003, Neurocomputing.

[30]  Yoshua Bengio,et al.  An empirical evaluation of deep architectures on problems with many factors of variation , 2007, ICML '07.

[31]  Thomas Serre,et al.  Robust Object Recognition with Cortex-Like Mechanisms , 2007, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[32]  Ying-Nong Chen,et al.  The Application of a Convolution Neural Network on Face and License Plate Detection , 2006, 18th International Conference on Pattern Recognition (ICPR'06).

[33]  Kunihiko Fukushima,et al.  Restoring partly occluded patterns: a neural network model , 2005, Neural Networks.

[34]  Glenn Carroll,et al.  On the Prospects for Building a Working Model of the Visual Cortex , 2007, AAAI.

[35]  P. Jonathon Phillips,et al.  Meta-Analysis of Third-Party Evaluations of Iris Recognition , 2007, 2007 First IEEE International Conference on Biometrics: Theory, Applications, and Systems.

[36]  Judea Pearl,et al.  Probabilistic reasoning in intelligent systems - networks of plausible inference , 1991, Morgan Kaufmann series in representation and reasoning.

[37]  Xin Yao,et al.  Evolving artificial neural network ensembles , 2008 .

[38]  Bogdan Kwolek,et al.  Face Detection Using Convolutional Neural Networks and Gabor Filters , 2005, ICANN.

[39]  Geoffrey E. Hinton,et al.  A time-delay neural network architecture for isolated word recognition , 1990, Neural Networks.

[40]  Yee Whye Teh,et al.  A Fast Learning Algorithm for Deep Belief Nets , 2006, Neural Computation.

[41]  Somsak Sukittanon,et al.  Convolutional networks for speech detection , 2004, INTERSPEECH.

[42]  H. Bülthoff,et al.  Learning to recognize objects , 1999, Trends in Cognitive Sciences.

[43]  Tai Sing Lee,et al.  Hierarchical Bayesian inference in the visual cortex. , 2003, Journal of the Optical Society of America. A, Optics, image science, and vision.

[44]  Thomas Hofmann,et al.  Greedy Layer-Wise Training of Deep Networks , 2007 .

[45]  Itamar Arel,et al.  DeSTIN: A Scalable Deep Learning Architecture with Application to High-Dimensional Robust Pattern Recognition , 2009, AAAI Fall Symposium: Biologically Inspired Cognitive Architectures.

[46]  Yann LeCun,et al.  Large-scale Learning with SVM and Convolutional for Generic Object Categorization , 2006, 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'06).

[47]  Yoshua. Bengio,et al.  Learning Deep Architectures for AI , 2007, Found. Trends Mach. Learn..

[48]  Hossein Mobahi,et al.  Deep learning from temporal coherence in video , 2009, ICML '09.

[49]  B. Ripley,et al.  Pattern Recognition , 1968, Nature.

[50]  Sven Behnke,et al.  Hierarchical Neural Networks for Image Interpretation , 2003, Lecture Notes in Computer Science.

[51]  Thomas Dean,et al.  A Computational Model of the Cerebral Cortex , 2005, AAAI.

[52]  Stephen V. Rice,et al.  The Fourth Annual Test of OCR Accuracy , 1995 .

[53]  緒方 淳,et al.  Real-time pedestrian detection using LIDAR and convolutional neural networks (特集 安全技術) , 2007 .

[54]  Yann LeCun,et al.  Deep belief net learning in a long-range vision system for autonomous off-road driving , 2008, 2008 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[55]  Honglak Lee,et al.  Unsupervised feature learning for audio classification using convolutional deep belief networks , 2009, NIPS.

[56]  R. Fergus,et al.  Learning invariant features through topographic filter maps , 2009, 2009 IEEE Conference on Computer Vision and Pattern Recognition.