Criticality & Deep Learning II: Momentum Renormalisation Group

Guided by critical systems found in nature we develop a novel mechanism consisting of inhomogeneous polynomial regularisation via which we can induce scale invariance in deep learning systems. Technically, we map our deep learning (DL) setup to a genuine field theory, on which we act with the Renormalisation Group (RG) in momentum space and produce the flow equations of the couplings; those are translated to constraints and consequently interpreted as "critical regularisation" conditions in the optimiser; the resulting equations hence prove to be sufficient conditions for - and serve as an elegant and simple mechanism to induce scale invariance in any deep learning setup.

[1]  E. M.,et al.  Statistical Mechanics , 2021, Manual for Theoretical Chemistry.

[2]  K. Wilson.  Renormalization Group and Critical Phenomena. I. Renormalization Group and the Kadanoff Scaling Picture , 1971 .

[3]  K. Wilson.  Renormalization Group and Critical Phenomena. II. Phase-Space Cell Analysis of Critical Behavior , 1971 .

[4]  K. Wilson.  The renormalization group and critical phenomena , 1983 .

[5]  Stuart A. Kauffman,et al.  The origins of order , 1993 .

[6]  R. Fox.  Review of Stuart Kauffman, The Origins of Order: Self-Organization and Selection in Evolution , 1993 .

[7]  Oono,et al.  Renormalization group and singular perturbations: Multiple scales, boundary layers, and reductive perturbation theory. , 1995, Physical review. E, Statistical physics, plasmas, fluids, and related interdisciplinary topics.

[8]  M. Fisher.  Renormalization group theory: Its basis and formulation in statistical physics , 1998 .

[9]  H. Stanley,et al.  Scaling, Universality, and Renormalization: Three Pillars of Modern Critical Phenomena , 1999 .

[10]  J. Finnigan.  How Nature Works; The science of self-organized criticality , 2001 .

[11]  J. Cardy.  Renormalization Methods---A Guide For Beginners , 2004 .

[12]  Michael J. Berry,et al.  Ising models for networks of real neurons , 2006, q-bio/0611072.

[13]  O. Isaeva,et al.  Approximate Description of the Mandelbrot Set. Thermodynamic Analogy , 2005, nlin/0504063.

[14]  D. Chialvo,et al.  The Brain: What is Critical About It? , 2008, 0804.0032.

[15]  O. Kogan,et al.  Renormalization group approach to oscillator synchronization. , 2008, Physical review. E, Statistical, nonlinear, and soft matter physics.

[16]  U. Täuber,et al.  Renormalization Group: Applications in Statistical Physics , 2011, 1112.1375.

[17]  W. Bialek,et al.  Are Biological Systems Poised at Criticality? , 2010, 1012.2242.

[18]  H. Nguyen.  Scale invariance in cosmology and physics , 2011, 1111.5529.

[19]  Martin S. Kochma'nski,et al.  Curie–Weiss magnet—a simple model of phase transition , 2013, 1301.2141.

[20]  David J. Schwab,et al.  An exact mapping between the Variational Renormalization Group and Deep Learning , 2014, ArXiv.

[21]  P. Hohenberg,et al.  An introduction to the Ginzburg-Landau theory of phase transitions and nonequilibrium patterns , 2014, 1410.7285.

[22]  Peter Toth,et al.  Criticality and Deep Learning, Part I: Theory vs. Empirics , 2017, ArXiv.

[23]  Adriano Barra,et al.  Phase transitions in Restricted Boltzmann Machines with generic priors , 2016, Physical review. E.