Deep self-organizing reservoir computing model for visual object recognition

Reservoir computing becomes increasingly a hot spot in recent years. In this paper, we propose a deep self-organizing reservoir computing model for visual object recognition. First, through combination of Kohonen's self-organizing map and SHESN network, we present a self-organizing SHESN (SO-SHESN). In the new model, we adopt the same mechanism of generating reservoir as SHESN, but McCulloch-Pitts type reservoir neuron is replaced with radial basis function neuron. Correspondingly, unsupervised competitive learning is exploited to train both input weights and reservoir weights of SO-SHESN. Second, we propose a deep SO-SHESN model through a stack of well-trained reservoir layers. In such a stacked structure, a novel trial-and-readout learning algorithm is used for pre-training of layer-wise reservoir, in which each layer is trained independently from each other. Finally, the experimental results obtained on MNIST benchmark dataset show that our SO-SHESN achieves the test recognition error rate of 5.66%, which improves classical ESN and SHESN by 6.44% and 1.74%, respectively. Furthermore, the test error rate of our deep SO-SHESN could reach up to 1.39%, which outperforms SO-SHESN with single reservoir layer by 4.27% and approximately approaches the state-of-the-art result of 1% among existing traditional machine learning approaches with non-CNN features.

[1]  Benjamin Schrauwen,et al.  Reservoir-based techniques for speech recognition , 2006, The 2006 IEEE International Joint Conference on Neural Network Proceedings.

[2]  Jason Weston,et al.  Natural Language Processing (Almost) from Scratch , 2011, J. Mach. Learn. Res..

[3]  Yee Whye Teh,et al.  A Fast Learning Algorithm for Deep Belief Nets , 2006, Neural Computation.

[4]  Peng Yu,et al.  Clustered complex echo state networks for traffic forecasting with prior knowledge , 2011, 2011 IEEE International Instrumentation and Measurement Technology Conference.

[5]  Yiannis Demiris,et al.  Spatio-Temporal Learning With the Online Finite and Infinite Echo-State Gaussian Processes , 2015, IEEE Transactions on Neural Networks and Learning Systems.

[6]  Dong Yu,et al.  Conversational Speech Transcription Using Context-Dependent Deep Neural Networks , 2012, ICML.

[7]  Pascal Vincent,et al.  Representation Learning: A Review and New Perspectives , 2012, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[8]  Terry Caelli,et al.  Learning Temporal Sequences in Recurrent Self-Organising Neural Nets , 1997, Australian Joint Conference on Artificial Intelligence.

[9]  Barbara Hammer,et al.  Recurrent networks for structured data – A unifying approach and its properties , 2002, Cognitive Systems Research.

[10]  Fangzheng Xue,et al.  A Priori Data-Driven Multi-Clustered Reservoir Generation Algorithm for Echo State Network , 2015, PloS one.

[11]  Herbert Jaeger,et al.  Optimization and applications of echo state networks with leaky- integrator neurons , 2007, Neural Networks.

[12]  Herbert Jaeger,et al.  The''echo state''approach to analysing and training recurrent neural networks , 2001 .

[13]  Stefan Rotter,et al.  Extending stability through hierarchical clusters in Echo State Networks , 2022 .

[14]  Sergey Ioffe,et al.  Rethinking the Inception Architecture for Computer Vision , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[15]  Yuichi Nakamura,et al.  Approximation of dynamical systems by continuous time recurrent neural networks , 1993, Neural Networks.

[16]  Henry Markram,et al.  Real-Time Computing Without Stable States: A New Framework for Neural Computation Based on Perturbations , 2002, Neural Computation.

[17]  Alessio Micheli,et al.  Recursive self-organizing network models , 2004, Neural Networks.

[18]  W S McCulloch,et al.  A logical calculus of the ideas immanent in nervous activity , 1990, The Philosophy of Artificial Intelligence.

[19]  Harald Haas,et al.  Harnessing Nonlinearity: Predicting Chaotic Systems and Saving Energy in Wireless Communication , 2004, Science.

[20]  Zhidong Deng,et al.  Collective Behavior of a Small-World Recurrent Neural System With Scale-Free Distribution , 2007, IEEE Transactions on Neural Networks.

[21]  Hermann Ney,et al.  Experiments with an extended tangent distance , 2000, Proceedings 15th International Conference on Pattern Recognition. ICPR-2000.

[22]  Herbert Jaeger,et al.  A tutorial on training recurrent neural networks , covering BPPT , RTRL , EKF and the " echo state network " approach - Semantic Scholar , 2005 .

[23]  Mantas Lukoševičius,et al.  On self-organizing reservoirs and their hierarchies , 2010 .

[24]  Fernando José Von Zuben,et al.  Self-organization and lateral interaction in echo state network reservoirs , 2014, Neurocomputing.

[25]  Yoshua Bengio,et al.  Modeling Temporal Dependencies in High-Dimensional Sequences: Application to Polyphonic Music Generation and Transcription , 2012, ICML.

[26]  Arthur E. Hoerl,et al.  Ridge Regression: Biased Estimation for Nonorthogonal Problems , 2000, Technometrics.

[27]  Samuel Kaski,et al.  Bibliography of Self-Organizing Map (SOM) Papers: 1981-1997 , 1998 .

[28]  Thomas Gärtner,et al.  A survey of kernels for structured data , 2003, SKDD.

[29]  Geoffrey E. Hinton,et al.  Generating Text with Recurrent Neural Networks , 2011, ICML.

[30]  Geoffrey E. Hinton,et al.  The Recurrent Temporal Restricted Boltzmann Machine , 2008, NIPS.

[31]  Jürgen Schmidhuber,et al.  Multi-column deep neural networks for image classification , 2012, 2012 IEEE Conference on Computer Vision and Pattern Recognition.

[32]  M M Merzenich,et al.  Temporal information transformed into a spatial code by a neural network with realistic properties , 1995, Science.

[33]  Yaochu Jin,et al.  Modeling neural plasticity in echo state networks for classification and regression , 2016, Inf. Sci..

[34]  Bill Triggs,et al.  Histograms of oriented gradients for human detection , 2005, 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05).

[35]  Thomas Voegtlin,et al.  Recursive self-organizing maps , 2002, Neural Networks.

[36]  Teuvo Kohonen,et al.  Self-organized formation of topologically correct feature maps , 2004, Biological Cybernetics.

[37]  Geoffrey E. Hinton,et al.  Learning internal representations by error propagation , 1986 .

[38]  Benjamin Schrauwen,et al.  An experimental unification of reservoir computing methods , 2007, Neural Networks.