Toward Idealized Decision Theory

This paper motivates the study of decision theory as necessary for aligning smarter-than-human artificial systems with human interests. We discuss the shortcomings of two standard formulations of decision theory, and demonstrate that they cannot be used to describe an idealized decision procedure suitable for approximation by artificial systems. We then explore the notions of policy selection and logical counterfactuals, two recent insights into decision theory that point the way toward promising paths for future research.

[1]  A. Wald.  Contributions to the Theory of Statistical Estimation and Testing Hypotheses , 1939 .

[2]  E. Rowland.  Theory of Games and Economic Behavior , 1946, Nature.

[3]  E. Lehmann.  Some Principles of the Theory of Testing Hypotheses , 1950 .

[4]  R. Jeffrey.  Ethics and the Logic of Decision , 1965 .

[5]  D. C. Cooper,et al.  Theory of Recursive Functions and Effective Computability , 1969, The Mathematical Gazette.

[6]  Robert Nozick,et al.  Newcomb’s Problem and Two Principles of Choice , 1969 .

[7]  M. Bar-Hillel,et al.  Newcomb's Paradox Revisited , 1972, The British Journal for the Philosophy of Science.

[8]  D. Rubin.  Estimating causal effects of treatments in randomized and nonrandomized studies. , 1974 .

[9]  William Harper,et al.  Counterfactuals and Two Kinds of Expected Utility , 1978 .

[10]  B. Skyrms.  Causal necessity: A pragmatic investigation of the necessity of laws , 1980 .

[11]  Douglas R. Hofstadter,et al.  Godel, Escher, Bach: An Eternal Golden Braid , 1981 .

[12]  David Lewis,et al.  Causal decision theory , 1981 .

[13]  E. Eells.  Metatickles and the dynamics of deliberation , 1984 .

[14]  E. Eells.  Causal Decision Theory , 1984, PSA: Proceedings of the Biennial Meeting of the Philosophy of Science Association.

[15]  T. Tan,et al.  The Bayesian foundations of solution concepts of games , 1988 .

[16]  Huw Price,et al.  Agency and Probabilistic Causality , 1991, The British Journal for the Philosophy of Science.

[17]  R. Gibbons.  Game theory for applied economists , 1992 .

[18]  D. Gauthier.  Assure and Threaten , 1994, Ethics.

[19]  Peter Norvig,et al.  Artificial Intelligence: A Modern Approach , 1995 .

[20]  James M. Joyce.  The Foundations of Causal Decision Theory , 1999 .

[21]  J. Pearl.  Causality: Models, Reasoning and Inference , 2000 .

[22]  James M. Joyce.  Levi on Causal Decision Theory and the Possibility of Predicting One's Own Actions , 2002 .

[23]  Jon Bird,et al.  The evolved radio and its implications for modelling the evolution of novel sensors , 2002, Proceedings of the 2002 Congress on Evolutionary Computation. CEC'02 (Cat. No.02TH8600).

[24]  A. Elga,et al.  Bayesianism, Infinite Decisions, and Binding , 2004 .

[25]  Simon Burgess,et al.  The Newcomb Problem: An Unqualified Resolution , 2004, Synthese.

[26]  H. Price.  Against causal decision theory , 1986, Synthese.

[27]  Arif Ahmed.  Evidential Decision Theory and Medical Newcomb Problems , 2005, The British Journal for the Philosophy of Science.

[28]  James M. Joyce.  Are Newcomb problems really decisions? , 2007, Synthese.

[29]  Eliezer Yudkowsky.  Artificial Intelligence as a Positive and Negative Factor in Global Risk , 2006 .

[30]  Frank Arntzenius,et al.  No regrets: or: Edith Piaf revamps decision theory , 2007, TARK '07.

[31]  A. Egan.  Some Counterexamples to Causal Decision Theory , 2007 .

[32]  Stephen M. Omohundro,et al.  The Basic AI Drives , 2008, AGI.

[33]  Christopher J. G. Meacham.  Binding and its consequences , 2010 .

[34]  Wolfgang Spohn,et al.  Reversing 30 years of discussion: why causal decision theorists should one-box , 2012, Synthese.

[35]  Ralph Wedgwood.  Gandalf’s solution to the Newcomb problem , 2011, Synthese.

[36]  J. Gustafsson.  A Note in Defence of Ratificationism , 2011 .

[37]  James M. Joyce.  Regret and instability in causal decision theory , 2012, Synthese.

[38]  Alex Altair,et al.  A Comparison of Decision Algorithms on Newcomblike Problems , 2023, ArXiv.

[39]  James Floyd Kelly,et al.  Push the Button , 2013 .

[40]  S. Brams,et al.  Prisoners' Dilemma is a Newcomb Problem , 2013 .

[41]  Nick Bostrom,et al.  Superintelligence: Paths, Dangers, Strategies , 2014 .

[42]  Benja Fallenstein,et al.  Program Equilibrium in the Prisoner ’ s Dilemma via Löb ’ s Theorem , 2014 .

[43]  Tsvi Benson-Tilsen,et al.  UDT WITH KNOWN SEARCH ORDER , 2014 .

[44]  Arif Ahmed.  Dicing with death , 2014 .

[45]  Arif Ahmed.  Infallibility in the Newcomb Problem , 2015 .

[46]  Benja Fallenstein,et al.  Aligning Superintelligence with Human Interests: A Technical Research Agenda , 2015 .