Simplifying ConvNets for Fast Learning

In this paper, we propose different strategies for simplifying filters, used as feature extractors, to be learnt in convolutional neural networks (ConvNets) in order to modify the hypothesis space, and to speed-up learning and processing times. We study two kinds of filters that are known to be computationally efficient in feed-forward processing: fused convolution/sub-sampling filters, and separable filters. We compare the complexity of the back-propagation algorithm on ConvNets based on these different kinds of filters. We show that using these filters allows to reach the same level of recognition performance as with classical ConvNets for handwritten digit recognition, up to 3.3 times faster.

[1]  Jirí Benes,et al.  On neural networks , 1990, Kybernetika.

[2]  J. L. Holt,et al.  Back propagation simulations using limited precision calculations , 1991, IJCNN-91-Seattle International Joint Conference on Neural Networks.

[3]  Russell Reed,et al.  Pruning algorithms-a survey , 1993, IEEE Trans. Neural Networks.

[4]  A. Petrowski.  Choosing among several parallel implementations of the backpropagation algorithm , 1994, Proceedings of 1994 IEEE International Conference on Neural Networks (ICNN'94).

[5]  Yoshua Bengio,et al.  Gradient-based learning applied to document recognition , 1998, Proc. IEEE.

[6]  Christophe Garcia,et al.  Convolutional face finder: a neural architecture for fast and robust face detection , 2004, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[7]  Yann LeCun,et al.  Synergistic Face Detection and Pose Estimation with Energy-Based Models , 2004, J. Mach. Learn. Res..

[8]  Patrice Y. Simard,et al.  High Performance Convolutional Neural Networks for Document Processing , 2006 .

[9]  Zohra Saidane,et al.  Automatic Scene Text Recognition using a Convolutional Neural Network , 2007 .

[10]  Christophe Garcia,et al.  Real-Time Video Convolutional Face Finder on Embedded Platforms , 2007, EURASIP J. Embed. Syst..

[11]  Stefan Duffner,et al.  Facial Image Processing with Convolutional Neural Networks , 2007 .

[12]  Yann LeCun,et al.  Deep belief net learning in a long-range vision system for autonomous off-road driving , 2008, 2008 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[13]  M. Kukacka,et al.  Hybrid convolutional neural networks , 2008, 2008 6th IEEE International Conference on Industrial Informatics.

[14]  Christophe Garcia,et al.  text Detection with Convolutional Neural Networks , 2008, VISAPP.

[15]  Yann LeCun,et al.  What is the best multi-stage architecture for object recognition? , 2009, 2009 IEEE 12th International Conference on Computer Vision.

[16]  Yann LeCun,et al.  Learning long‐range vision for autonomous off‐road driving , 2009, J. Field Robotics.

[17]  Christophe Garcia,et al.  Embedded facial image processing with Convolutional Neural Networks , 2010, Proceedings of 2010 IEEE International Symposium on Circuits and Systems.

[18]  Luca Maria Gambardella,et al.  Handwritten Digit Recognition with a Committee of Deep Neural Nets on GPUs , 2011, ArXiv.

[19]  Tapani Raiko,et al.  Deep Learning Made Easier by Linear Transformations in Perceptrons , 2012, AISTATS.