Combining additive input noise annealing and pattern transformations for improved handwritten character recognition

Abstract Two problems that burden the learning process of Artificial Neural Networks with Back Propagation are the need of building a full and representative learning data set, and the avoidance of stalling in local minima. Both problems seem to be closely related when working with the handwritten digits contained in the MNIST dataset. Using a modest sized ANN, the proposed combination of input data transformations enables the achievement of a test error as low as 0.43%, which is up to standard compared to other more complex neural architectures like Convolutional or Deep Neural Networks.

[1]  Sargur N. Srihari,et al.  On-Line and Off-Line Handwriting Recognition: A Comprehensive Survey , 2000, IEEE Trans. Pattern Anal. Mach. Intell..

[2]  C. P. Unsworth,et al.  Excessive Noise Injection Training of Neural Networks for Markerless Tracking in Obscured and Segmented Environments , 2006, Neural Computation.

[3]  Guozhong An,et al.  The Effects of Adding Noise During Backpropagation Training on a Generalization Performance , 1996, Neural Computation.

[4]  Kunihiko Fukushima,et al.  Neocognitron: A self-organizing neural network model for a mechanism of pattern recognition unaffected by shift in position , 1980, Biological Cybernetics.

[5]  Adam P. Piotrowski,et al.  Comparison of evolutionary computation techniques for noise injected neural network training to estimate longitudinal dispersion coefficients in rivers , 2012, Expert Syst. Appl..

[6]  Charles B. Owen,et al.  Statistical analysis of the effect of noise injection during neural network training , 1993, Defense, Security, and Sensing.

[7]  Yves Grandvalet,et al.  Noise Injection: Theoretical Prospects , 1997, Neural Computation.

[8]  Luca Maria Gambardella,et al.  Convolutional Neural Network Committees for Handwritten Character Classification , 2011, 2011 International Conference on Document Analysis and Recognition.

[9]  Yann LeCun,et al.  The mnist database of handwritten digits , 2005 .

[10]  Patrice Y. Simard,et al.  Best practices for convolutional neural networks applied to visual document analysis , 2003, Seventh International Conference on Document Analysis and Recognition, 2003. Proceedings..

[11]  Luca Maria Gambardella,et al.  Deep Big Simple Neural Nets Excel on Handwritten Digit Recognition , 2010, ArXiv.

[12]  Jocelyn Sietsma,et al.  Creating artificial neural networks that generalize , 1991, Neural Networks.

[13]  Geoffrey E. Hinton.  Learning multiple layers of representation , 2007, Trends in Cognitive Sciences.

[14]  Petri Koistinen,et al.  Using additive noise in back-propagation training , 1992, IEEE Trans. Neural Networks.

[15]  Luca Maria Gambardella,et al.  Better Digit Recognition with a Committee of Simple Neural Nets , 2011, 2011 International Conference on Document Analysis and Recognition.

[16]  D. Hubel,et al.  Receptive fields, binocular interaction and functional architecture in the cat's visual cortex , 1962, The Journal of physiology.

[17]  Araceli Sanchis,et al.  A Skeletonizing Reconfigurable Self-Organizing Model: Validation Through Text Recognition , 2011, Neural Processing Letters.

[18]  Jürgen Schmidhuber,et al.  Multi-column deep neural networks for image classification , 2012, 2012 IEEE Conference on Computer Vision and Pattern Recognition.

[19]  George D. C. Cavalcanti,et al.  AutoAssociative Pyramidal Neural Network for one class pattern classification with implicit feature extraction , 2013, Expert Syst. Appl..

[20]  Richard F. Lyon,et al.  Effective Training of a Neural Network Character Classifier for Word Recognition , 1996, NIPS.

[21]  Araceli Sanchis,et al.  Modified Self-organizing Maps for Line Extraction in Digitized Text Documents , 2003, Applied Informatics.

[22]  Petri Koistinen,et al.  Kernel regression and backpropagation training with noise , 1991, [Proceedings] 1991 IEEE International Joint Conference on Neural Networks.

[23]  Chuan Wang,et al.  Training neural networks with additive noise in the desired signal , 1999, IEEE Trans. Neural Networks.

[24]  Christopher M. Bishop,et al.  Current address: Microsoft Research, , 2022 .

[25]  Geoffrey E. Hinton,et al.  Learning a Nonlinear Embedding by Preserving Class Neighbourhood Structure , 2007, AISTATS.

[26]  Carlos A. B. Mello,et al.  Segmentation of connected handwritten digits using Self-Organizing Maps , 2013, Expert Syst. Appl..

[27]  Sepp Hochreiter,et al.  The Vanishing Gradient Problem During Learning Recurrent Neural Nets and Problem Solutions , 1998, Int. J. Uncertain. Fuzziness Knowl. Based Syst..

[28]  Kunihiko Fukushima,et al.  Neocognitron: A Self-Organizing Neural Network Model for a Mechanism of Visual Pattern Recognition , 1982 .

[29]  Thomas Hofmann,et al.  Greedy Layer-Wise Training of Deep Networks , 2007 .

[30]  Verónica Bolón-Canedo,et al.  Toward the scalability of neural networks through feature selection , 2013, Expert Syst. Appl..

[31]  Antonio Fernández Caballero,et al.  Display text segmentation after learning best-fitted ocr binarization parameters , 2012 .

[32]  Kiyotoshi Matsuoka,et al.  Noise injection into inputs in back-propagation learning , 1992, IEEE Trans. Syst. Man Cybern..

[33]  Luca Maria Gambardella,et al.  Deep, Big, Simple Neural Nets for Handwritten Digit Recognition , 2010, Neural Computation.

[34]  Yann LeCun,et al.  What is the best multi-stage architecture for object recognition? , 2009, 2009 IEEE 12th International Conference on Computer Vision.

[35]  Marc'Aurelio Ranzato,et al.  Efficient Learning of Sparse Representations with an Energy-Based Model , 2006, NIPS.

[36]  PlamondonRéjean,et al.  On-Line and Off-Line Handwriting Recognition , 2000 .

[37]  Yoshua Bengio,et al.  Gradient-based learning applied to document recognition , 1998, Proc. IEEE.