Thermodynamic cost and benefit of data representations

This paper takes a thermodynamic approach to addressing the problem of how to represent data efficiently and meaningfully, a problem at the heart of learning and adaptation in both biological and artificial systems. Thermodynamic analysis of an information engine's cyclic operation reveals information theoretic quantities that are setting limits on performance. If run at fixed temperature, dissipation is lower bounded by a term proportional to irrelevant information. Data representation strategies that are optimal in the sense of minimizing dissipation must therefore strive to retain only relevant information. When an information engine is allowed to make use of a temperature difference, it can produce net work output, for which an upper bound is derived. Maximizing the bound yields a direct derivation of the Information Bottleneck method, a known technique in signal processing and machine learning, used precisely to filter relevant information from irrelevant clutter.

[1]  R. Allers.  Cybernetics. Control and Communication in the Animal and the Machine by Norbert Wiener (review) , 2017 .

[2]  L. Brillouin.  Maxwell's Demon Cannot Operate: Information and Entropy. I , 1951 .

[3]  E. Jaynes.  Information Theory and Statistical Mechanics , 1957 .

[4]  L. Szilard.  On the decrease of entropy in a thermodynamic system by the intervention of intelligent beings. , 1964, Behavioral science.

[5]  Toby Berger,et al.  Rate distortion theory : a mathematical basis for data compression , 1971 .

[6]  Robert Shaw,et al.  The Dripping Faucet As A Model Chaotic System , 1984 .

[7]  Geoffrey E. Hinton,et al.  A Learning Algorithm for Boltzmann Machines , 1985, Cogn. Sci..

[8]  Editors , 1986, Brain Research Bulletin.

[9]  Ralph Linsker,et al.  Self-organization in a perceptual network , 1988, Computer.

[10]  Thomas M. Cover,et al.  Elements of Information Theory , 2005 .

[11]  Aaron D. Wyner,et al.  Coding Theorems for a Discrete Source With a Fidelity CriterionInstitute of Radio Engineers, International Convention Record, vol. 7, 1959. , 1993 .

[12]  Terrence J. Sejnowski,et al.  Blind separation and blind deconvolution: an information-theoretic approach , 1995, 1995 International Conference on Acoustics, Speech, and Signal Processing.

[13]  Seth Lloyd,et al.  Quantum-mechanical Maxwell’s demon , 1997 .

[14]  M..  Szilard ' s heat engine , 1999 .

[15]  Naftali Tishby,et al.  The information bottleneck method , 2000, ArXiv.

[16]  Charles H. Bennett,et al.  Notes on Landauer's Principle, Reversible Computation, and Maxwell's Demon , 2002, physics/0210005.

[17]  William Bialek,et al.  How Many Clusters? An Information-Theoretic Perspective , 2003, Neural Computation.

[18]  Sang Joon Kim,et al.  A Mathematical Theory of Communication , 2006 .

[19]  G. Crooks.  Beyond Boltzmann-Gibbs statistics: maximum entropy hyperensembles out of equilibrium. , 2006, Physical review. E, Statistical, nonlinear, and soft matter physics.

[20]  Susanne Still,et al.  Information-theoretic approach to interactive learning , 2007, 0709.1948.

[21]  Dean J. Driebe,et al.  Generalization of the second law for a transition between nonequilibrium states , 2010 .

[22]  Takahiro Sagawa,et al.  Quantum Szilard engine. , 2010, Physical review letters.

[23]  Modeling Maxwell's demon with a microcanonical Szilard engine. , 2011, Physical review. E, Statistical, nonlinear, and soft matter physics.

[24]  E. Lutz,et al.  Experimental verification of Landauer’s principle linking information and thermodynamics , 2012, Nature.

[25]  Susanne Still,et al.  The thermodynamics of prediction , 2012, Physical review letters.

[26]  Christopher Jarzynski,et al.  Work and information processing in a solvable model of Maxwell’s demon , 2012, Proceedings of the National Academy of Sciences.

[27]  W. Bialek.  Biophysics: Searching for Principles , 2012 .

[28]  Masahito Ueda,et al.  Nonequilibrium thermodynamics of feedback control. , 2011, Physical review. E, Statistical, nonlinear, and soft matter physics.

[29]  Arne L. Grimsmo,et al.  Quantum correlations in predictive processes , 2013, 1302.5552.

[30]  Susanne Still,et al.  Information Bottleneck Approach to Predictive Inference , 2014, Entropy.

[31]  Yonggun Jun,et al.  High-precision test of Landauer's principle in a feedback trap. , 2014, Physical review letters.

[32]  Experimental study of mutual information in a Maxwell Demon , 2014, 1405.1272.

[33]  Zhiyue Lu,et al.  Engineering Maxwell's demon , 2014 .

[34]  David Jennings,et al.  Description of quantum coherence in thermodynamic processes requires constraints beyond free energy , 2014, Nature Communications.

[35]  T. Sagawa,et al.  Thermodynamics of information , 2015, Nature Physics.

[36]  Susanne Still,et al.  LOSSY IS LAZY , 2015 .

[37]  Pieter Rein ten Wolde,et al.  Thermodynamics of Computational Copying in Biochemical Systems , 2015, 1503.00909.

[38]  A. B. Boyd,et al.  Maxwell Demon Dynamics: Deterministic Chaos, the Szilard Map, and the Intelligence of Thermodynamic Systems. , 2015, Physical review letters.

[39]  Neri Merhav.  Relations Between Work and Entropy Production for General Information-Driven, Finite-State Engines , 2016, ArXiv.

[40]  D. Petrov,et al.  Brownian Carnot engine , 2014, Nature Physics.

[41]  John Bechhoefer,et al.  Erasure without Work in an Asymmetric Double-Well Potential. , 2016, Physical review letters.

[42]  Eld,et al.  Identifying functional thermodynamics in autonomous Maxwellian ratchets , 2016 .

[43]  Sebastian Deffner,et al.  Quantum work and the thermodynamic cost of quantum measurements. , 2016, Physical review. E.

[44]  John Bechhoefer,et al.  Brownian duet: A novel tale of thermodynamic efficiency , 2016, 1607.04388.

[45]  Momčilo Gavrilov,et al.  Direct measurement of weakly nonequilibrium system entropy is consistent with Gibbs–Shannon form , 2017, Proceedings of the National Academy of Sciences.