Duality Between Learning Machines: A Bridge Between Supervised and Unsupervised Learning

We exhibit a duality between two perceptrons that allows us to compare the theoretical analysis of supervised and unsupervised learning tasks. The first perceptron has one output and is asked to learn a classification of p patterns. The second (dual) perceptron has p outputs and is asked to transmit as much information as possible on a distribution of inputs. We show in particular that the maximum information that can be stored in the couplings for the supervised learning task is equal to the maximum information that can be transmitted by the dual perceptron.

[1]  Vladimir Vapnik,et al.  Chervonenkis: On the uniform convergence of relative frequencies of events to their probabilities , 1971 .

[2]  Deco,et al.  Statistical-ensemble theory of redundancy reduction and the duality between unsupervised and supervised neural learning. , 1995, Physical review. E, Statistical physics, plasmas, fluids, and related interdisciplinary topics.

[3]  Vladimir Vapnik,et al.  Estimation of Dependences Based on Empirical Data: Springer Series in Statistics (Springer Series in Statistics) , 1982 .

[4]  Robert O. Winder,et al.  Enumeration of Seven-Argument Threshold Functions , 1965, IEEE Trans. Electron. Comput..

[5]  J J Hopfield,et al.  Neural networks and physical systems with emergent collective computational abilities. , 1982, Proceedings of the National Academy of Sciences of the United States of America.

[6]  P. Peretto.  An introduction to the modeling of neural networks , 1992 .

[7]  E. Reed,et al.  The ecological approach to learning , 1981, Behavioral and Brain Sciences.

[8]  B. V. Limaye.  Uniform convergence , 2018, The Student Mathematical Library.

[9]  Teuvo Kohonen,et al.  Self-organization and associative memory: 3rd edition , 1989 .

[10]  M. Mézard,et al.  Spin Glass Theory and Beyond , 1987 .

[11]  J. Nadal,et al.  Information capacity of a perceptron , 1992 .

[12]  W. Krauth,et al.  Storage capacity of memory networks with binary couplings , 1989 .

[13]  Rémi Monasson,et al.  Properties of neural networks storing spatially correlated patterns , 1992 .

[14]  Richard E. Blahut,et al.  Principles and practice of information theory , 1987 .

[15]  H. Gutfreund,et al.  Capacity of neural networks with discrete synaptic couplings , 1990 .

[16]  Anders Krogh,et al.  Introduction to the theory of neural computation , 1994, The advanced book program.

[17]  Ralph Linsker,et al.  Self-organization in a perceptual network , 1988, Computer.

[18]  Néstor Parga,et al.  Information Processing by a Perceptron , 1992, Int. J. Neural Syst..

[19]  Néstor Parga,et al.  Information processing by a perceptron in an unsupervised learning task , 1993 .

[20]  J. Nadal,et al.  From statistical physics to statistical inference and back , 1994 .

[21]  E. Gardner.  The space of interactions in neural network models , 1988 .

[22]  Esther Levin,et al.  A statistical approach to learning and generalization in layered neural networks , 1989, Proc. IEEE.

[23]  Zee,et al.  Understanding the efficiency of human perception. , 1988, Physical review letters.

[24]  Teuvo Kohonen,et al.  Self-Organization and Associative Memory , 1988 .

[25]  Thomas M. Cover,et al.  Geometrical and Statistical Properties of Systems of Linear Inequalities with Applications in Pattern Recognition , 1965, IEEE Trans. Electron. Comput..