Bio-inspired problems in rate-distortion theory

Designed and evolved sensors are faced with a difficult optimization problem: storing more information about the environment and reaping greater “rewards” costs time, energy, and material. Quantifying the ben- efits and costs associated with storing environmental information will aid design of more efficient sensors and allow other researchers to test whether or not a particular biosensor is, in fact, efficient. One principled approach to this quantification comes from rate-distortion theory. We discuss three biologically inspired problems in rate-distortion theory. First, we validate a new approach to perceptually compressing natural image patches; sec- ond, we show that sensing tradeoffs in large fluctuating environments depend only on coarse environmental statistics; and third, we describe a new method for calculating predictive features and memory-prediction trade- offs in stochastic environments. For the last of these problems, we suggest a new class of stimuli that might be used to profitably test whether a par- ticular biosensor is an efficient predictor of its environment.

[1]  William Bialek,et al.  Neural Coding of Natural Stimuli: Information at Sub-Millisecond Resolution , 2007, BMC Neuroscience.

[2]  Karoline Wiesner,et al.  A New Method for Inferring Hidden Markov Models from Noisy Time Sequences , 2012, PloS one.

[3]  Sharmishtha Mitra,et al.  Theory of Point Estimation - Web course , 2000 .

[4]  Michael J. Berry,et al.  Predictive information in a sensory population , 2013, Proceedings of the National Academy of Sciences.

[5]  H. B. Barlow,et al.  Possible Principles Underlying the Transformations of Sensory Messages , 2012 .

[6]  Tomáš Gedeon,et al.  Bifurcation Structure of a Class of SN-invariant Constrained Optimization Problems , 2004 .

[7]  S. Laughlin.  Energy as a constraint on the coding and processing of sensory information , 2001, Current Opinion in Neurobiology.

[8]  James P. Crutchfield,et al.  Information Symmetries in Irreversible Processes , 2011, Chaos.

[9]  D. Luenberger.  Optimization by Vector Space Methods , 1968 .

[10]  Inderjit S. Dhillon,et al.  An information theoretic analysis of maximum likelihood mixture estimation for exponential families , 2004, ICML.

[11]  Nicholas F. Travers.  Exponential Bounds for Convergence of Entropy Rate Approximations in Hidden Markov Models Satisfying a Path-Mergeability Condition , 2012, 1211.6181.

[12]  Alan C. Bovik,et al.  Mean squared error: Love it or leave it? A new look at Signal Fidelity Measures , 2009, IEEE Signal Processing Magazine.

[13]  Raymond W. Yeung,et al.  Information Theory and Network Coding , 2008 .

[14]  William B. Levy,et al.  Energy Efficient Neural Codes , 1996, Neural Computation.

[15]  Rob R. de Ruyter van Steveninck,et al.  The metabolic cost of neural information , 1998, Nature Neuroscience.

[16]  Daniel Ray Upper,et al.  Theory and algorithms for hidden Markov models and generalized hidden Markov models , 1998 .

[17]  S. S. Melnik,et al.  Entropy and long-range correlations in DNA sequences , 2014, Comput. Biol. Chem..

[18]  A. M. Walker.  On the Asymptotic Behaviour of Posterior Distributions , 1969 .

[19]  J. H. van Hateren,et al.  Modelling the Power Spectra of Natural Images: Statistics and Information , 1996, Vision Research.

[20]  Kenneth Rose,et al.  A mapping approach to rate-distortion computation and analysis , 1994, IEEE Trans. Inf. Theory.

[21]  Takuma Akimoto,et al.  Characterization of intermittency in renewal processes: application to earthquakes. , 2010, Physical review. E, Statistical, nonlinear, and soft matter physics.

[22]  R. Williams,et al.  The control of neuron number. , 1988, Annual review of neuroscience.

[23]  Chun-Biu Li,et al.  Multiscale complex network of protein conformational fluctuations in single-molecule time series , 2008, Proceedings of the National Academy of Sciences.

[24]  Stephanie E. Palmer,et al.  Optimal prediction and natural scene statistics in the retina , 2015, 1507.00125.

[25]  Masahito Ueda,et al.  Minimal energy cost for thermodynamic information processing: measurement and information erasure. , 2008, Physical review letters.

[26]  Jascha Sohl-Dickstein,et al.  Efficient and optimal binary Hopfield associative memory storage using minimum probability flow , 2012, 1204.2916.

[27]  Tomás Gedeon,et al.  Annealing and the Rate Distortion Problem , 2002, NIPS.

[28]  Tsvi Tlusty,et al.  A model for the emergence of the genetic code as a transition in a noisy information channel , 2007, Journal of theoretical biology.

[29]  Pablo A. Iglesias,et al.  An Information-Theoretic Characterization of the Optimal Gradient Sensing Response of Cells , 2007, PLoS Comput. Biol..

[30]  Chris R Sims,et al.  The cost of misremembering: Inferring the loss function in visual working memory. , 2015, Journal of vision.

[31]  James P. Crutchfield,et al.  Bayesian Structural Inference for Hidden Processes , 2013, Physical review. E, Statistical, nonlinear, and soft matter physics.

[32]  James P. Crutchfield,et al.  Anatomy of a Bit: Information in a Time Series Observation , 2011, Chaos.

[33]  Aaron D. Wyner,et al.  Coding Theorems for a Discrete Source With a Fidelity CriterionInstitute of Radio Engineers, International Convention Record, vol. 7, 1959. , 1993 .

[34]  James P. Crutchfield,et al.  Many Roads to Synchrony: Natural Time Scales and Their Algorithms , 2010, Physical review. E, Statistical, nonlinear, and soft matter physics.

[35]  Robert B. Gramacy,et al.  A brief history of long memory , 2014 .

[36]  James P. Crutchfield,et al.  Equivalence of History and Generator-Machines , 2012 .

[37]  Eero P. Simoncelli,et al.  Natural image statistics and neural representation. , 2001, Annual review of neuroscience.

[38]  James P. Crutchfield,et al.  Computational Mechanics: Pattern and Prediction, Structure and Simplicity , 1999, ArXiv.

[39]  Larry Wasserman,et al.  Asymptotic Properties of Nonparametric Bayesian Procedures , 1998 .

[40]  Lav R. Varshney,et al.  Optimal Information Storage in Noisy Synapses under Resource Constraints , 2006, Neuron.

[41]  Pooneh Mohammadiara.  The Elusive Present: Hidden Past and Future Correlation and Why We Build Models , 2014 .

[42]  Ruby C. Weng,et al.  Asymptotic posterior normality for multiparameter problems , 2008 .

[43]  Joseph J. Atick,et al.  Towards a Theory of Early Visual Processing , 1990, Neural Computation.

[44]  Naftali Tishby,et al.  Complexity through nonextensivity , 2001, physics/0103076.

[45]  Wulfram Gerstner,et al.  Neuronal Dynamics: From Single Neurons To Networks And Models Of Cognition , 2014 .

[46]  Sergio Verdú,et al.  Fixed-Length Lossy Compression in the Finite Blocklength Regime , 2011, IEEE Transactions on Information Theory.

[47]  Naftali Tishby,et al.  The information bottleneck method , 2000, ArXiv.

[48]  D. Blackwell,et al.  On the Identifiability Problem for Functions of Finite Markov Chains , 1957 .

[49]  Richard S. Sutton,et al.  Reinforcement Learning: An Introduction , 1998, IEEE Trans. Neural Networks.

[50]  Michael J. Berry,et al.  Metabolically Efficient Information Processing , 2001, Neural Computation.

[51]  S. Laughlin,et al.  An Energy Budget for Signaling in the Grey Matter of the Brain , 2001, Journal of cerebral blood flow and metabolism : official journal of the International Society of Cerebral Blood Flow and Metabolism.

[52]  An Unsymmetric Fubini Theorem , 1984 .

[53]  Eero P. Simoncelli,et al.  Image quality assessment: from error visibility to structural similarity , 2004, IEEE Transactions on Image Processing.

[54]  H. Risken.  The Fokker-Planck equation : methods of solution and applications , 1985 .

[55]  Douglas Lind,et al.  An Introduction to Symbolic Dynamics and Coding , 1995 .

[56]  A. Fisher,et al.  The Theory of critical phenomena , 1992 .

[57]  A. J. Bell.  THE CO-INFORMATION LATTICE , 2003 .

[58]  William Bialek,et al.  Spikes: Exploring the Neural Code , 1996 .

[59]  Jascha Sohl-Dickstein,et al.  A new method for parameter estimation in probabilistic models: Minimum probability flow , 2011, Physical review letters.

[60]  James P. Crutchfield,et al.  Computational Mechanics of Input-Output Processes: Structured transformations and the ε-transducer , 2014, ArXiv.

[61]  Hans G. Feichtinger,et al.  Analysis, Synthesis, and Estimation of Fractal-Rate Stochastic Point Processes , 1997, adap-org/9709006.

[62]  Paolo Grigolini,et al.  Brain, music, and non-Poisson renewal processes. , 2006, Physical review. E, Statistical, nonlinear, and soft matter physics.

[63]  James P. Crutchfield,et al.  Structure or Noise? , 2007, ArXiv.

[64]  D. J. Daley,et al.  The Hurst Index of Long-Range Dependent Renewal Processes , 1999 .

[65]  Simon B. Laughlin,et al.  Energy-Efficient Coding with Discrete Stochastic Events , 2002, Neural Computation.

[66]  Stephen E. Levinson,et al.  Continuously variable duration hidden Markov models for automatic speech recognition , 1986 .

[67]  Joel G. Smith,et al.  The Information Capacity of Amplitude- and Variance-Constrained Scalar Gaussian Channels , 1971, Inf. Control..

[68]  J J Hopfield,et al.  Neural networks and physical systems with emergent collective computational abilities. , 1982, Proceedings of the National Academy of Sciences of the United States of America.

[69]  Susanne Still,et al.  Optimal causal inference: estimating stored information and approximating causal architecture. , 2007, Chaos.

[70]  William Bialek,et al.  How Many Clusters? An Information-Theoretic Perspective , 2003, Neural Computation.

[71]  Susanne Still,et al.  Information Bottleneck Approach to Predictive Inference , 2014, Entropy.

[72]  T. Berger.  Rate-Distortion Theory , 2003 .

[73]  James P. Crutchfield,et al.  Chaos Forgets and Remembers: Measuring Information Creation, Destruction, and Storage , 2013, ArXiv.

[74]  Michael Brand,et al.  Proliferation, neurogenesis and regeneration in the non-mammalian vertebrate brain , 2008, Philosophical Transactions of the Royal Society B: Biological Sciences.

[75]  Raymond W. Yeung,et al.  The Interplay Between Entropy and Variational Distance , 2007, IEEE Transactions on Information Theory.

[76]  H Barlow,et al.  Redundancy reduction revisited , 2001, Network.