Information, Utility and Bounded Rationality

Perfectly rational decision-makers maximize expected utility, but crucially ignore the resource costs incurred when determining optimal actions. Here we employ an axiomatic framework for bounded rational decision-making based on a thermodynamic interpretation of resource costs as information costs. This leads to a variational "free utility" principle akin to thermodynamical free energy that trades off utility and information costs. We show that bounded optimal control solutions can be derived from this variational principle, which leads in general to stochastic policies. Furthermore, we show that risk-sensitive and robust (minimax) control schemes fall out naturally from this framework if the environment is considered as a bounded rational and perfectly rational opponent, respectively. When resource costs are ignored, the maximum expected utility principle is recovered.

[1]  J. Neumann,et al.  Theory of games and economic behavior , 1945, 100 Years of Math Milestones.

[2]  M. Tribus,et al.  Energy and information , 1971 .

[3]  H. Simon,et al.  Models of Bounded Rationality: Empirically Grounded Economic Reason , 1997 .

[4]  R. Callen,et al.  Thermodynamics and an Introduction to Thermostatistics, 2nd Edition , 1985 .

[5]  P. Whittle.  Risk-Sensitive Optimal Control , 1990 .

[6]  Tamer Başar,et al.  H1-Optimal Control and Related Minimax Design Problems , 1995 .

[7]  Anthony J. G. Hey,et al.  Feynman Lectures on Computation , 1996 .

[8]  T. Basar,et al.  H∞-0ptimal Control and Related Minimax Design Problems: A Dynamic Game Approach , 1996, IEEE Trans. Autom. Control..

[9]  D. Haussler,et al.  MUTUAL INFORMATION, METRIC ENTROPY AND CUMULATIVE RELATIVE ENTROPY RISK , 1997 .

[10]  G. Keller.  Equilibrium States in Ergodic Theory , 1998 .

[11]  H. Crauel.  EQUILIBRIUM STATES IN ERGODIC THEORY (London Mathematical Society Student Texts 42) By G ERHARD K ELLER : 178 pp., £16.95 (LMS members' price £12.71), ISBN 0-521-59534-7 (Cambridge University Press, 1998). , 2001 .

[12]  David H. Wolpert,et al.  Information Theory - The Bridge Connecting Bounded Rational Game Theory and Statistical Physics , 2004, ArXiv.

[13]  H. Kappen.  Linear theory for control of nonlinear stochastic systems. , 2004, Physical review letters.

[14]  Paul M. B. Vitányi.  Time, space, and energy in reversible computing , 2005, CF '05.

[15]  Emanuel Todorov,et al.  Linearly-solvable Markov decision problems , 2006, NIPS.

[16]  Daniel A. Braun,et al.  A conversion between utility and information , 2009, AGI 2010.

[17]  Emanuel Todorov,et al.  Efficient computation of optimal actions , 2009, Proceedings of the National Academy of Sciences.

[18]  Susanne Still,et al.  Information-theoretic approach to interactive learning , 2007, 0709.1948.

[19]  Leon G. Higley,et al.  Forensic Entomology: An Introduction , 2009 .

[20]  Daniel A. Braun,et al.  A Bayesian Rule for Adaptive Control based on Causal Interventions , 2009, AGI 2010.

[21]  Daniel A. Braun,et al.  A Minimum Relative Entropy Principle for Adaptive Control in Linear Quadratic Regulators , 2010, ICINCO.

[22]  Daniel A. Braun,et al.  A Minimum Relative Entropy Principle for Learning and Acting , 2008, J. Artif. Intell. Res..

[23]  Stefan Schaal,et al.  Path integral control and bounded rationality , 2011, 2011 IEEE Symposium on Adaptive Dynamic Programming and Reinforcement Learning (ADPRL).