Optimization theory of Hebbian/anti-Hebbian networks for PCA and whitening

In analyzing information streamed by sensory organs, our brains face challenges similar to those solved in statistical signal processing. This suggests that biologically plausible implementations of online signal processing algorithms may model neural computation. Here, we focus on such workhorses of signal processing as Principal Component Analysis (PCA) and whitening which maximize information transmission in the presence of noise. We adopt the similarity matching framework, recently developed for principal subspace extraction, but modify the existing objective functions by adding a decorrelating term. From the modified objective functions, we derive online PCA and whitening algorithms which are implementable by neural networks with local learning rules, i.e. synaptic weight updates that depend on the activity of only pre- and postsynaptic neurons. Our theory offers a principled model of neural computations and makes testable predictions such as the dropout of underutilized neurons.

[1]  Tao Hu,et al.  A Hebbian/Anti-Hebbian network for online sparse dictionary learning derived from symmetric matrix factorization , 2014, 2014 48th Asilomar Conference on Signals, Systems and Computers.

[2]  Joel Zylberberg,et al.  Inhibitory Interneurons Decorrelate Excitatory Cells to Drive Sparse Code Formation in a Spiking Model of V1 , 2013, The Journal of Neuroscience.

[3]  Dmitri B. Chklovskii,et al.  A Normative Theory of Adaptive Dimensionality Reduction in Neural Networks , 2015, NIPS.

[4]  Ralph Linsker,et al.  Self-organization in a perceptual network , 1988, Computer.

[5]  David J. Field,et al.  Sparse coding with an overcomplete basis set: A strategy employed by V1? , 1997, Vision Research.

[6]  Christopher J. Rozell,et al.  Modeling Inhibitory Interneurons in Efficient Sensory Coding Models , 2015, PLoS Comput. Biol..

[7]  Sun-Yuan Kung,et al.  A neural network learning algorithm for adaptive principal component extraction (APEX) , 1990, International Conference on Acoustics, Speech, and Signal Processing.

[8]  Gilad Lerman,et al.  Robust Stochastic Principal Component Analysis , 2014, AISTATS.

[9]  Juha Karhunen,et al.  Principal component neural networks — Theory and applications , 1998, Pattern Analysis and Applications.

[10]  Mark D. Plumbley.  A Subspace Network That Determines Its Own Output Dimension , 1994 .

[11]  Stephen P. Boyd,et al.  Convex Optimization , 2004, Algorithms and Theory of Computation Handbook.

[12]  Dmitri B. Chklovskii,et al.  A Hebbian/Anti-Hebbian network derived from online non-negative matrix factorization can cluster and discover sparse features , 2014, 2014 48th Asilomar Conference on Signals, Systems and Computers.

[13]  Joseph J. Atick,et al.  What Does the Retina Know about Natural Scenes? , 1992, Neural Computation.

[14]  M D Plumbley.  Information processing in negative feedback neural networks. , 1996, Network.

[15]  A. Koulakov,et al.  Sparse Incomplete Representations: A Potential Role of Olfactory Granule Cells , 2011, Neuron.

[16]  Aapo Hyvärinen,et al.  Natural Image Statistics - A Probabilistic Approach to Early Computational Vision , 2009, Computational Imaging and Vision.

[17]  E. Oja.  Simplified neuron model as a principal component analyzer , 1982, Journal of mathematical biology.

[18]  Tao Hu,et al.  A Hebbian/Anti-Hebbian Neural Network for Linear Subspace Learning: A Derivation from Multidimensional Scaling of Streaming Data , 2015, Neural Computation.

[19]  Nitish Srivastava,et al.  Dropout: a simple way to prevent neural networks from overfitting , 2014, J. Mach. Learn. Res..

[20]  Tao Hu,et al.  A mechanistic model of early sensory processing based on subtracting sparse representations , 2012, NIPS.

[21]  Todd K. Leen.  Dynamics of Learning in Recurrent Feature-Discovery Networks , 1990, NIPS.

[22]  N. L. Johnson,et al.  Multivariate Analysis , 1958, Nature.

[23]  Tao Hu,et al.  A neuron as a signal processing device , 2013, 2013 Asilomar Conference on Signals, Systems and Computers.

[24]  Koby Crammer,et al.  Online Tracking of Linear Subspaces , 2006, COLT.

[25]  Nathan Srebro,et al.  Stochastic optimization for PCA and PLS , 2012, 2012 50th Annual Allerton Conference on Communication, Control, and Computing (Allerton).

[26]  J. Rubner,et al.  A Self-Organizing Network for Principal-Component Analysis , 1989 .

[27]  Erkki Oja,et al.  Principal components, minor components, and linear neural networks , 1992, Neural Networks.

[28]  Mark D. Plumbley.  A Hebbian/anti-Hebbian network which optimizes information capacity by orthonormalizing the principal subspace , 1993 .

[29]  P. Foldiak,et al.  Adaptive network for optimal linear feature extraction , 1989, International 1989 Joint Conference on Neural Networks.

[30]  Terence D. Sanger,et al.  Optimal unsupervised learning in a single-layer linear feedforward neural network , 1989, Neural Networks.