Whence the Expected Free Energy?

The expected free energy (EFE) is a central quantity in the theory of active inference. It is the quantity that all active inference agents are mandated to minimize through action, and its decomposition into extrinsic and intrinsic value terms is key to the balance of exploration and exploitation that active inference agents evince. Despite its importance, the mathematical origins of this quantity and its relation to the variational free energy (VFE) remain unclear. In this letter, we investigate the origins of the EFE in detail and show that it is not simply ”the free energy in the future.” We present a functional that we argue is the natural extension of the VFE but actively discourages exploratory behavior, thus demonstrating that exploration does not directly follow from free energy minimization into the future. We then develop a novel objective, the free energy of the expected future (FEEF), which possesses both the epistemic component of the EFE and an intuitive mathematical grounding as the divergence between predicted and desired futures.

[1]  Taweh Beysolow II.  Mathematical Review , 2020, Microwave and Wireless Synthesizers.

[2]  Beren Millidge,et al.  On the Relationship Between Active Inference and Control as Inference , 2020, IWAI.

[3]  Anil K. Seth,et al.  Reinforcement Learning through Active Inference , 2020, ArXiv.

[4]  Tim Verbelen,et al.  Learning Perception and Planning With Deep Active Inference , 2020, ICASSP 2020 - 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[5]  Karl J. Friston,et al.  Active inference on discrete state-spaces: A synthesis , 2020, Journal of mathematical psychology.

[6]  Alexander Tschantz,et al.  Scaling Active Inference , 2019, 2020 International Joint Conference on Neural Networks (IJCNN).

[7]  Beren Millidge,et al.  Deep Active Inference as Variational Policy Gradients , 2019, Journal of Mathematical Psychology.

[8]  Karl J. Friston,et al.  Markov blankets, information geometry and stochastic thermodynamics , 2019, Philosophical Transactions of the Royal Society A.

[9]  Karl J. Friston,et al.  Generalised free energy and active inference , 2018, Biological Cybernetics.

[10]  Anil K. Seth,et al.  Learning action-oriented models through active inference , 2019, bioRxiv.

[11]  Karl J. Friston.  A free energy principle for a particular physics , 2019, 1906.10184.

[12]  Bert de Vries,et al.  Simulating Active Inference Processes by Message Passing , 2019, Frontiers Robotics AI.

[13]  Thijs van de Laar,et al.  Simulating Active Inference Processes by Message Passing , 2019, Front. Robot. AI.

[14]  Beren Millidge.  Implementing Predictive Processing and Active Inference: Preliminary Steps and Results , 2019 .

[15]  Beren Millidge.  Combining Active Inference and Hierarchical Predictive Coding: A Tutorial Introduction and Case Study , 2019 .

[16]  Karl J. Friston,et al.  Neuronal message passing using Mean-field, Bethe, and Marginal approximations , 2019, Scientific Reports.

[17]  Karl J. Friston,et al.  Impulsivity and Active Inference , 2019, Journal of Cognitive Neuroscience.

[18]  Alexei A. Efros,et al.  Large-Scale Study of Curiosity-Driven Learning , 2018, ICLR.

[19]  Thomas Parr,et al.  The computational neurology of active vision , 2019 .

[20]  Kai Ueltzhöffer,et al.  Deep active inference , 2017, Biological Cybernetics.

[21]  Karl J. Friston,et al.  Computational mechanisms of curiosity and goal-directed exploration , 2018, bioRxiv.

[22]  Karl J. Friston,et al.  Active Inference in OpenAI Gym: A Paradigm for Computational Investigations Into Psychiatric Illness. , 2018, Biological psychiatry. Cognitive neuroscience and neuroimaging.

[23]  Stefan J. Kiebel,et al.  Active Inference, Belief Propagation, and the Bethe Approximation , 2018, Neural Computation.

[24]  Christoph Salge,et al.  Expanding the Active Inference Landscape: More Intrinsic Motivations in the Perception-Action Loop , 2018, Front. Neurorobot..

[25]  Sergey Levine,et al.  Reinforcement Learning and Control as Probabilistic Inference: Tutorial and Review , 2018, ArXiv.

[26]  Christopher L. Buckley,et al.  A probabilistic interpretation of PID controllers using active inference , 2018, bioRxiv.

[27]  Karl J. Friston,et al.  Active inference and the anatomy of oculomotion , 2018, Neuropsychologia.

[28]  Karl J. Friston,et al.  The Computational Anatomy of Visual Neglect , 2017, Cerebral cortex.

[29]  Karl J. Friston,et al.  The graphical brain: Belief propagation and active inference , 2017, Network Neuroscience.

[30]  Karl J. Friston,et al.  Uncertainty, epistemics and active inference , 2017, Journal of The Royal Society Interface.

[31]  Karl J. Friston,et al.  Active Inference, Curiosity and Insight , 2017, Neural Computation.

[32]  Karl J. Friston,et al.  The active construction of the visual world , 2017, Neuropsychologia.

[33]  Christopher L. Buckley,et al.  An active inference implementation of phototaxis , 2017, ECAL.

[34]  Karl J. Friston,et al.  Deep temporal models and active inference , 2017, Neuroscience and biobehavioral reviews.

[35]  Karl J. Friston,et al.  Predicting green: really radical (plant) predictive processing , 2017, Journal of The Royal Society Interface.

[36]  Simon McGregor,et al.  The free energy principle for action and perception: A mathematical review , 2017, 1705.09156.

[37]  Alexei A. Efros,et al.  Curiosity-Driven Exploration by Self-Supervised Prediction , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[38]  Evangelos A. Theodorou,et al.  Model Predictive Path Integral Control: From Theory to Parallel Computation , 2017 .

[39]  Karl J. Friston,et al.  Active Inference: A Process Theory , 2017, Neural Computation.

[40]  Stewart Shipp,et al.  Neural Elements for Predictive Coding , 2016, Front. Psychol..

[41]  Karl J. Friston,et al.  Neuroscience and Biobehavioral Reviews , 2022 .

[42]  Karl J. Friston,et al.  Active Inference, epistemic value, and vicarious trial and error , 2016, Learning & memory.

[43]  Karl J. Friston,et al.  Scene Construction, Visual Foraging, and Active Inference , 2016, Front. Comput. Neurosci..

[44]  David M. Blei,et al.  Variational Inference: A Review for Statisticians , 2016, ArXiv.

[45]  J. Schulman,et al.  Variational Information Maximizing Exploration , 2016 .

[46]  Shakir Mohamed,et al.  Variational Information Maximisation for Intrinsically Motivated Reinforcement Learning , 2015, NIPS.

[47]  Karl J. Friston,et al.  Cerebral hierarchies: predictive processing, precision and the pulvinar , 2015, Philosophical Transactions of the Royal Society B: Biological Sciences.

[48]  Karl J. Friston,et al.  Active inference and epistemic value , 2015, Cognitive neuroscience.

[49]  Raymond J. Dolan,et al.  Active Inference, Evidence Accumulation, and the Urn Task , 2015, Neural Computation.

[50]  Sophie Denève,et al.  Bayesian Inference with Spiking Neurons , 2004, Encyclopedia of Computational Neuroscience.

[51]  K. Rawlik.  On probabilistic inference approaches to stochastic optimal control , 2013 .

[52]  Raymond J. Dolan,et al.  Exploration, novelty, surprise, and free energy minimization , 2013, Front. Psychol..

[53]  Evangelos Theodorou,et al.  Relative entropy and free energy dualities: Connections to Path Integral and KL control , 2012, 2012 IEEE 51st IEEE Conference on Decision and Control (CDC).

[54]  Karl J. Friston,et al.  Canonical Microcircuits for Predictive Coding , 2012, Neuron.

[55]  Stefan J. Kiebel,et al.  Evidence for neural encoding of Bayesian surprise in human somatosensation , 2012, NeuroImage.

[56]  Marc Toussaint,et al.  On Stochastic Optimal Control and Reinforcement Learning by Approximate Inference , 2012, Robotics: Science and Systems.

[57]  Karl J. Friston,et al.  Free Energy, Value, and Attractors , 2011, Comput. Math. Methods Medicine.

[58]  Stephen J. Roberts,et al.  A tutorial on variational Bayesian inference , 2012, Artificial Intelligence Review.

[59]  Doina Precup,et al.  An information-theoretic approach to curiosity-driven reinforcement learning , 2012, Theory in Biosciences.

[60]  Karl J. Friston.  What Is Optimal about Motor Control? , 2011, Neuron.

[61]  Yi Sun,et al.  Planning to Be Surprised: Optimal Bayesian Exploration in Dynamic Environments , 2011, AGI.

[62]  Karl J. Friston,et al.  Action understanding and active inference , 2011, Biological Cybernetics.

[63]  Pierre Baldi,et al.  Of bits and wows: A Bayesian theory of surprise with applications to attention , 2010, Neural Networks.

[64]  Stefan Schaal,et al.  A Generalized Path Integral Control Approach to Reinforcement Learning , 2010, J. Mach. Learn. Res..

[65]  Karl J. Friston.  The free-energy principle: a unified brain theory? , 2010, Nature Reviews Neuroscience.

[66]  Karl J. Friston,et al.  Reinforcement Learning or Active Inference? , 2009, PloS one.

[67]  Pierre Baldi,et al.  Bayesian surprise attracts human attention , 2005, Vision Research.

[68]  Marc Toussaint,et al.  Probabilistic inference as a model of planned behavior , 2009, Künstliche Intell..

[69]  Michael I. Jordan,et al.  Graphical Models, Exponential Families, and Variational Inference , 2008, Found. Trends Mach. Learn..

[70]  Karl J. Friston.  Hierarchical Models in the Brain , 2008, PLoS Comput. Biol..

[71]  Michael W. Spratling.  Reconciling Predictive Coding and Biased Competition Models of Cortical Function , 2008, Frontiers Comput. Neurosci..

[72]  Karl J. Friston.  Variational filtering , 2008, NeuroImage.

[73]  Karl J. Friston,et al.  DEM: A variational treatment of dynamic systems , 2008, NeuroImage.

[74]  Pierre-Yves Oudeyer,et al.  What is Intrinsic Motivation? A Typology of Computational Approaches , 2007, Frontiers Neurorobotics.

[75]  H. Kappen.  An introduction to stochastic control theory, path integrals and reinforcement learning , 2007 .

[76]  Rajesh P. N. Rao,et al.  Bayesian brain : probabilistic approaches to neural coding , 2006 .

[77]  Karl J. Friston,et al.  A free energy principle for the brain , 2006, Journal of Physiology-Paris.

[78]  Tutut Herawan,et al.  Computational and mathematical methods in medicine. , 2006, Computational and mathematical methods in medicine.

[79]  William T. Freeman,et al.  Constructing free-energy approximations and generalized belief propagation algorithms , 2005, IEEE Transactions on Information Theory.

[80]  H. Kappen.  Path integrals and symmetry breaking for optimal control theory , 2005, physics/0505066.

[81]  Karl J. Friston,et al.  A theory of cortical responses , 2005, Philosophical Transactions of the Royal Society B: Biological Sciences.

[82]  D. Knill,et al.  The Bayesian brain: the role of uncertainty in neural coding and computation , 2004, Trends in Neurosciences.

[83]  Amos Storkey,et al.  Advances in Neural Information Processing Systems 20 , 2007 .

[84]  Karl J. Friston.  Learning and inference in the brain , 2003, Neural Networks.

[85]  Hagai Attias,et al.  Planning by Probabilistic Inference , 2003, AISTATS.

[86]  Matthew J. Beal.  Variational algorithms for approximate Bayesian inference , 2003 .

[87]  W. Freeman,et al.  Generalized Belief Propagation , 2000, NIPS.