Learning the pseudoinverse solution to network weights

The last decade has seen the parallel emergence in computational neuroscience and machine learning of neural network structures which spread the input signal randomly to a higher dimensional space; perform a nonlinear activation; and then solve for a regression or classification output by means of a mathematical pseudoinverse operation. In the field of neuromorphic engineering, these methods are increasingly popular for synthesizing biologically plausible neural networks, but the "learning method"-computation of the pseudoinverse by singular value decomposition-is problematic both for biological plausibility and because it is not an online or an adaptive method. We present an online or incremental method of computing the pseudoinverse precisely, which we argue is biologically plausible as a learning method, and which can be made adaptable for non-stationary data streams. The method is significantly more memory-efficient than the conventional computation of pseudoinverses by singular value decomposition.

[1]  Y. Takane,et al.  Generalized Inverse Matrices , 2011 .

[2]  Yann LeCun,et al.  The mnist database of handwritten digits , 2005 .

[3]  Teuvo Kohonen,et al.  Self-organization and associative memory: 3rd edition , 1989 .

[4]  T. Greville,et al.  Some Applications of the Pseudoinverse of a Matrix , 1960 .

[5]  Mattia Rigotti,et al.  A Simple Derivation of a Bound on the Perceptron Margin Using Singular Value Decomposition , 2011, Neural Computation.

[6]  Heekuck Oh,et al.  Neural Networks for Pattern Recognition , 1993, Adv. Comput..

[7]  Steve B. Furber,et al.  Real time on-chip implementation of dynamical systems with spiking neurons , 2012, The 2012 International Joint Conference on Neural Networks (IJCNN).

[8]  Kwabena Boahen,et al.  Silicon Neurons That Compute , 2012, ICANN.

[9]  Narasimhan Sundararajan,et al.  On-Line Sequential Extreme Learning Machine , 2005, Computational Intelligence.

[10]  Xiao-Jing Wang,et al.  Internal Representation of Task Rules by Recurrent Dynamics: The Importance of the Diversity of Neural Responses , 2010, Front. Comput. Neurosci..

[11]  C. Eliasmith,et al.  Learning to Select Actions with Spiking Neurons in the Basal Ganglia , 2012, Front. Neurosci..

[12]  Teuvo Kohonen,et al.  Self-Organizing Maps , 2010 .

[13]  Michael R. Lyu,et al.  A pseudoinverse learning algorithm for feedforward neural networks with stacked generalization applications to software reliability growth data , 2004, Neurocomputing.

[14]  Adi Ben-Israel,et al.  Generalized inverses: theory and applications , 1974 .

[15]  L. Glass,et al.  Oscillation and chaos in physiological control systems. , 1977, Science.

[16]  Teuvo Kohonen,et al.  Self-Organization and Associative Memory , 1988 .

[17]  Pavel Kovanic.  On the pseudoinverse of a sum of symmetric matrices with applications to estimation , 1979, Kybernetika.

[18]  Erkki Oja,et al.  Neural Networks, Principal Components, and Subspaces , 1989, Int. J. Neural Syst..

[19]  Teuvo Kohonen,et al.  Correlation Matrix Memories , 1972, IEEE Transactions on Computers.

[20]  Pat Langley,et al.  Editorial: On Machine Learning , 1986, Machine Learning.

[21]  Chris Eliasmith,et al.  Neural Engineering: Computation, Representation, and Dynamics in Neurobiological Systems , 2004, IEEE Transactions on Neural Networks.

[22]  Yoshua Bengio,et al.  Gradient-based learning applied to document recognition , 1998, Proc. IEEE.

[23]  Zhenghao Chen,et al.  On Random Weights and Unsupervised Feature Learning , 2011, ICML.

[24]  Eero P. Simoncelli,et al.  Natural signal statistics and sensory gain control , 2001, Nature Neuroscience.

[25]  Chris Eliasmith,et al.  Fine-Tuning and the Stability of Recurrent Neural Networks , 2011, PloS one.

[26]  Lawrence K. Saul,et al.  Large-Margin Classification in Infinite Neural Networks , 2010, Neural Computation.

[27]  R. O’Reilly.  Six principles for biologically based computational models of cortical cognition , 1998, Trends in Cognitive Sciences.

[28]  AI Koan.  Weighted Sums of Random Kitchen Sinks : Replacing minimization with randomization in learning , 2008 .

[29]  Chee Kheong Siew,et al.  Extreme learning machine: Theory and applications , 2006, Neurocomputing.