What decision theory provides the best procedure for identifying the best action available to a given artificially intelligent system

Decision theory has had a long-standing history in the behavioural and social sciences as a tool for constructing good approximations of human behaviour. Yet as artificially intelligent systems (AIs) grow in intellectual capacity and eventually outpace humans, decision theory becomes evermore important as a model of AI behaviour. What sort of decision procedure might an AI employ? In this work, I propose that policy-based causal decision theory (PCDT), which places a primacy on the decision-relevance of predictors and simulations of agent behaviour, may be such a procedure. I compare this account to the recently-developed functional decision theory (FDT), which is motivated by similar concerns. I also address potentially counterintuitive features of PCDT, such as its refusal to condition on observations made at certain times.

[1]  G. Drescher.  Good and Real: Demystifying Paradoxes from Physics to Ethics , 2006 .

[2]  D. Gauthier.  XII—In the Neighbourhood of the Newcomb-Predictor (Reflections on Rationality) , 1989 .

[3]  Arif Ahmed,et al.  Evidence, Decision and Causality , 2014 .

[4]  Stephen M. Omohundro,et al.  The Basic AI Drives , 2008, AGI.

[5]  Nate Soares,et al.  A Formal Approach to the Problem of Logical Non-Omniscience , 2017, TARK.

[6]  Brian Skyrms,et al.  The Dynamics Of Rational Deliberation , 1990 .

[7]  E. McClennen.  Rationality and Dynamic Choice: Foundational Explorations , 1996 .

[8]  E. Eells.  Causal Decision Theory , 1984, PSA: Proceedings of the Biennial Meeting of the Philosophy of Science Association.

[9]  Ken Binmore,et al.  The Theory of Choice: A Critical Guide , 1992 .

[10]  Frank Arntzenius,et al.  No regrets: or: Edith Piaf revamps decision theory , 2007, TARK '07.

[11]  A. Egan.  Some Counterexamples to Causal Decision Theory , 2007 .

[12]  B. Levinstein,et al.  Cheating Death in Damascus , 2020, The Journal of Philosophy.

[13]  Preston Greene.  Success-First Decision Theories , 2018, Newcomb's Problem.

[14]  C. Pollard,et al.  Center for the Study of Language and Information , 2022 .

[15]  Christopher J. G. Meacham.  Binding and its consequences , 2010 .

[16]  R. Wedgwood,et al.  Instrumental Rationality , 2010, ArgMAS.

[17]  D. Gauthier.  Assure and Threaten , 1994, Ethics.

[18]  Judea Pearl,et al.  CAUSATION, ACTION, AND COUNTERFACTUALS , 2004 .

[19]  Finn V. Jensen,et al.  Bayesian Networks and Decision Graphs , 2001, Statistics for Engineering and Information Science.

[20]  Ned Hall,et al.  Causation and counterfactuals , 2004 .

[21]  Wolfgang Spohn,et al.  Reversing 30 years of discussion: why causal decision theorists should one-box , 2012, Synthese.

[22]  Preston Greene.  Rationality and success , 2013 .

[23]  P. Suppes.  A Probabilistic Theory Of Causality , 1970 .

[24]  James M. Joyce.  Causal reasoning and backtracking , 2009 .

[25]  C. Aitken,et al.  The logic of decision , 2014 .

[26]  Rachael Briggs,et al.  Normative Theories of Rational Choice: Expected Utility , 2014 .

[27]  Daniel Hintze.  Problem Class Dominance in Predictive Dilemmas , 2014 .

[28]  Robert Nozick,et al.  Newcomb’s Problem and Two Principles of Choice , 1969 .

[29]  Nate Soares,et al.  Functional Decision Theory: A New Theory of Instrumental Rationality , 2017, ArXiv.

[30]  Ralph Wedgwood.  Gandalf’s solution to the Newcomb problem , 2011, Synthese.

[31]  Ellery Eells.  Rational Decision and Causality , 1982 .

[32]  A. Baier,et al.  Reasons and Persons , 1984 .

[33]  William Harper,et al.  Counterfactuals and Two Kinds of Expected Utility , 1978 .

[34]  Robert Stalnaker.  Letter to David Lewis , 1980 .

[35]  Gualtiero Piccinini,et al.  Physical computation : a mechanistic account , 2015 .

[36]  Tom Burr,et al.  Causation, Prediction, and Search , 2003, Technometrics.

[37]  J. Schreiber.  Foundations Of Statistics , 2016 .

[38]  David J. Chalmers,et al.  On implementing a computation , 1994, Minds and Machines.

[39]  James M. Joyce.  Regret and instability in causal decision theory , 2012, Synthese.

[40]  S. Bernstein.  Omission impossible , 2016 .

[41]  S. Brams,et al.  Prisoners' Dilemma is a Newcomb Problem , 2013 .

[42]  James M. Joyce.  The Foundations of Causal Decision Theory , 1999 .

[43]  E. Eells.  Metatickles and the dynamics of deliberation , 1984 .

[44]  Paul Weirich,et al.  Realistic Decision Theory: Rules for Nonideal Agents in Nonideal Circumstances , 2004 .

[45]  Tsvi Benson-Tilsen,et al.  UDT WITH KNOWN SEARCH ORDER , 2014 .

[46]  A. Morton.  Epistemic Virtues, Metavirtues, and Computational Complexity , 2004 .

[47]  B. Skyrms.  Causal necessity: A pragmatic investigation of the necessity of laws , 1980 .

[48]  Daniel Cohen,et al.  On What Cannot Be , 1990 .