Neural Networks Retrieving Boolean Patterns in a Sea of Gaussian Ones

Restricted Boltzmann machines are key tools in machine learning and are described by the energy function of bipartite spin-glasses. From a statistical mechanical perspective, they share the same Gibbs measure of Hopfield networks for associative memory. In this equivalence, weights in the former play as patterns in the latter. As Boltzmann machines usually require real weights to be trained with gradient-descent-like methods, while Hopfield networks typically store binary patterns to be able to retrieve, the investigation of a mixed Hebbian network, equipped with both real (e.g., Gaussian) and discrete (e.g., Boolean) patterns naturally arises. We prove that, in the challenging regime of a high storage of real patterns, where retrieval is forbidden, an additional load of Boolean patterns can still be retrieved, as long as the ratio between the overall load and the network size does not exceed a critical threshold, that turns out to be the same of the standard Amit–Gutfreund–Sompolinsky theory. Assuming replica symmetry, we study the case of a low load of Boolean patterns combining the stochastic stability and Hamilton-Jacobi interpolating techniques. The result can be extended to the high load by a non rigorous but standard replica computation argument.

[1]  O. L. Z..  Book Review: The Organization of Behaviour: A Neuropsychological Theory , 1950 .

[2]  S. Varadhan.  Large Deviations and Applications , 1984 .

[3]  Geoffrey E. Hinton,et al.  A Learning Algorithm for Boltzmann Machines , 1985, Cogn. Sci..

[4]  Sompolinsky,et al.  Storing infinite numbers of patterns in a spin-glass model of neural networks. , 1985, Physical review letters.

[5]  Sompolinsky,et al.  Spin-glass models of neural networks. , 1985, Physical review. A, General physics.

[6]  M. Mézard,et al.  Spin Glass Theory and Beyond , 1987 .

[7]  Daniel J. Amit,et al.  Modeling brain function: the world of attractor neural networks, 1st Edition , 1989 .

[8]  D. J. Wallace,et al.  Training with noise and the storage of correlated patterns in a neural network model , 1989 .

[9]  Anders Krogh,et al.  Introduction to the theory of neural computation , 1994, The advanced book program.

[10]  Sompolinsky,et al.  Statistical mechanics of learning from examples. , 1992, Physical review. A, Atomic, molecular, and optical physics.

[11]  A. Bovier,et al.  The retrieval phase of the Hopfield model: A rigorous analysis of the overlap distribution* , 1995, cond-mat/9507111.

[12]  A. Bovier,et al.  Gibbs states of the Hopfield model with extensively many patterns , 1995 .

[13]  A. Bovier,et al.  Metastates in the Hopfield Model in the Replica Symmetric Regime , 1997, cond-mat/9706080.

[14]  A. Bovier,et al.  Hopfield Models as Generalized Random Mean Field Models , 1996, cond-mat/9607103.

[15]  M. Talagrand.  Rigorous results for the Hopfield model with many patterns , 1998 .

[16]  M. Aizenman,et al.  On the Stability of the Quenched State in Mean-Field Spin-Glass Models , 1997, cond-mat/9712129.

[17]  A. Bovier,et al.  Stochastic Symmetry-Breaking in a Gaussian Hopfield Model , 1998, cond-mat/9807086.

[18]  John J. Hopfield,et al.  Neural networks and physical systems with emergent collective computational abilities , 1999 .

[19]  M. Talagrand.  Exponential inequalities and convergence of moments in the replica-symmetric regime of the Hopfield model , 2000 .

[20]  M. Shcherbina,et al.  Rigorous Solution of the Gardner Problem , 2001, math-ph/0112003.

[21]  R. C. Tees.  Review of The organization of behavior: A neuropsychological theory. , 2003 .

[22]  P. Carmona,et al.  Universality in Sherrington–Kirkpatrick's spin glass model , 2004, math/0403359.

[23]  Mariya Shcherbina.  Some Mathematical Problems of Neural Networks Theory , 2005 .

[24]  Peter Sollich,et al.  Theory of Neural Information Processing Systems , 2005 .

[25]  Yee Whye Teh,et al.  A Fast Learning Algorithm for Deep Belief Nets , 2006, Neural Computation.

[26]  A. Barra.  The Mean Field Ising Model trough Interpolating Techniques , 2007, 0712.1344.

[27]  A. Barra,et al.  About the ergodic regime in the analogical Hopfield neural networks: Moments of the partition function , 2008, 0911.3515.

[28]  Geoffrey E. Hinton,et al.  Deep Boltzmann Machines , 2009, AISTATS.

[29]  A. Barra,et al.  The Replica Symmetric Approximation of the Analogical Neural Network , 2009, 0911.3096.

[30]  A. Barra,et al.  Replica symmetry breaking in mean-field spin glasses through the Hamilton–Jacobi technique , 2010, 1003.5226.

[31]  A. Barra,et al.  Equilibrium statistical mechanics of bipartite spin systems , 2010, 1012.1261.

[32]  A. Barra,et al.  About a solvable mean field model of a Gaussian spin glass , 2011, 1109.4069.

[33]  P. Contucci,et al.  Perspectives on Spin Glasses , 2012 .

[34]  A. Barra,et al.  Interpolating the Sherrington–Kirkpatrick replica trick , 2011, 1104.2080.

[35]  Razvan Pascanu,et al.  Learning Algorithms for the Classification Restricted Boltzmann Machine , 2012, J. Mach. Learn. Res..

[36]  Adriano Barra,et al.  On the equivalence of Hopfield networks and Boltzmann Machines , 2011, Neural Networks.

[37]  G. Genovese.  Universality in bipartite mean field spin glasses , 2011, 1102.2535.

[38]  Elena Agliari,et al.  Multitasking associative networks. , 2011, Physical review letters.

[39]  A. Barra,et al.  How glassy are neural networks? , 2012, 1205.3900.

[40]  A. Barra,et al.  Mean field spin glasses treated with PDE techniques , 2013 .

[41]  Elena Agliari,et al.  Parallel retrieval of correlated patterns: From Hopfield networks to Boltzmann machines , 2013, Neural Networks.

[42]  P. Tiberto,et al.  Magnetic properties of jet-printer inks containing dispersed magnetite nanoparticles , 2013 .

[43]  A. Barra,et al.  Multi-Species Mean Field Spin Glasses. Rigorous Results , 2013, 1307.5154.

[44]  A. Barra,et al.  Mean field bipartite spin models treated with mechanical techniques , 2013, 1310.5901.

[45]  Antonio Auffinger,et al.  Free Energy and Complexity of Spherical Bipartite Models , 2014, 1405.2321.

[46]  P. Baldi,et al.  Searching for exotic particles in high-energy physics with deep learning , 2014, Nature Communications.

[47]  Adriano Barra,et al.  Extensive parallel processing on scale-free networks. , 2014, Physical review letters.

[48]  Florent Krzakala,et al.  Training Restricted Boltzmann Machines via the Thouless-Anderson-Palmer Free Energy , 2015, NIPS 2015.

[49]  Yann LeCun,et al.  The Loss Surfaces of Multilayer Networks , 2014, AISTATS.

[50]  A. Barra,et al.  Retrieval capabilities of hierarchical networks: from Dyson to Hopfield. , 2015, Physical review letters.

[51]  Michael Levin,et al.  Inferring Regulatory Networks from Experimental Morphological Phenotypes: A Computational Method Reverse-Engineers Planarian Regeneration , 2015, PLoS Comput. Biol..

[52]  G. Genovese,et al.  Non-Convex Multipartite Ferromagnets , 2015, 1511.01307.

[53]  Haiping Huang,et al.  Advanced Mean Field Theory of Restricted Boltzmann Machine , 2015, Physical review. E, Statistical, nonlinear, and soft matter physics.

[54]  Guigang Zhang,et al.  Deep Learning , 2016, Int. J. Semantic Comput..

[55]  Haiping Huang,et al.  Unsupervised feature learning from finite data by message passing: discontinuous versus continuous phase transition , 2016, Physical review. E.

[56]  Haiping Huang,et al.  Statistical mechanics of unsupervised feature learning in a restricted Boltzmann machine with binary synapses , 2016, ArXiv.

[57]  Rémi Monasson,et al.  Emergence of Compositional Representations in Restricted Boltzmann Machines , 2016, Physical review letters.

[58]  M. Mézard.  Mean-field message-passing equations in the Hopfield model and its generalizations. , 2016, Physical review. E.

[59]  Adriano Barra,et al.  Phase transitions in Restricted Boltzmann Machines with generic priors , 2016, Physical review. E.

[60]  Adriano Barra,et al.  Phase Diagram of Restricted Boltzmann Machines and Generalised Hopfield Networks with Arbitrary Priors , 2017, Physical review. E.