Human-aligned artificial intelligence is a multiobjective problem

As the capabilities of artificial intelligence (AI) systems improve, it becomes important to constrain their actions to ensure their behaviour remains beneficial to humanity. A variety of ethical, legal and safety-based frameworks have been proposed as a basis for designing these constraints. Despite their variations, these frameworks share the common characteristic that decision-making must consider multiple potentially conflicting factors. We demonstrate that these alignment frameworks can be represented as utility functions, but that the widely used Maximum Expected Utility (MEU) paradigm provides insufficient support for such multiobjective decision-making. We show that a Multiobjective Maximum Expected Utility paradigm based on the combination of vector utilities and non-linear action–selection can overcome many of the issues which limit MEU’s effectiveness in implementing aligned AI. We examine existing approaches to multiobjective AI, and identify how these can contribute to the development of human-aligned intelligent agents.

[1]  David A. Ferrucci,et al.  Introduction to "This is Watson" , 2012, IBM J. Res. Dev..

[2]  Shimon Whiteson,et al.  A Survey of Multi-Objective Sequential Decision-Making , 2013, J. Artif. Intell. Res..

[3]  H. Tavani,et al.  Ethics and technology : controversies, questions, and strategies for ethical computing , 2010 .

[4]  Daniel Angus,et al.  Multiple objective ant colony optimisation , 2009, Swarm Intelligence.

[5]  Mark O. Riedl,et al.  Using Stories to Teach Human Values to Artificial Agents , 2016, AAAI Workshop: AI, Ethics, and Society.

[6]  Peter Danielson.  Can robots have a conscience? , 2009, Nature.

[7]  F. Cushman.  Action, Outcome, and Value , 2013, Personality and social psychology review : an official journal of the Society for Personality and Social Psychology, Inc.

[8]  Michael P. Wellman.  REASONING ABOUT PREFERENCE MODELS , 1985 .

[9]  J. Altmann.  Arms control for armed uninhabited vehicles: an ethical issue , 2013, Ethics and Information Technology.

[10]  D. Dubois,et al.  Beyond min aggregation in multicriteria decision: (ordered) weighted min, discri-min, leximin , 1997 .

[11]  Noel E. Sharkey,et al.  Death strikes from the sky: the calculus of proportionality , 2009, IEEE Technology and Society Magazine.

[12]  Roman V. Yampolskiy,et al.  Artificial Intelligence Safety and Cybersecurity: a Timeline of AI Failures , 2016, ArXiv.

[13]  Ronald Leenes,et al.  Laws on Robots, Laws by Robots, Laws in Robots: Regulating Robot Behaviour by Design , 2014 .

[14]  Michael Anderson,et al.  MedEthEx: A Prototype Medical Ethics Advisor , 2006, AAAI.

[15]  Nick Bostrom,et al.  Thinking Inside the Box: Controlling and Using an Oracle AI , 2012, Minds and Machines.

[16]  C. Allen,et al.  Moral Machines: Teaching Robots Right from Wrong , 2008 .

[17]  Peter Vamplew,et al.  Lego Mindstorms Robots as a Platform for Teaching Reinforcement Learning , 2004 .

[18]  Demis Hassabis,et al.  Mastering the game of Go with deep neural networks and tree search , 2016, Nature.

[19]  Ralph L. Keeney.  Value-driven expert systems for decision support , 1987 .

[20]  Mariarosaria Taddeo,et al.  The ethics of algorithms: Mapping the debate , 2016, Big Data Soc..

[21]  Peter Asaro,et al.  A Body to Kick, but Still No Soul to Damn : Legal Perspectives on Robotics , 2012 .

[22]  Patrick Lin,et al.  Moral Machines: Contradiction in Terms or Abdication of Human Responsibility? , 2012 .

[23]  George Reynolds.  Ethics in Information Technology , 2002 .

[24]  Shane Legg,et al.  Human-level control through deep reinforcement learning , 2015, Nature.

[25]  Ioannis P. Vlahavas,et al.  Multiobjective heuristic state-space planning , 2003, Artif. Intell..

[26]  Michael L. Littman,et al.  Reinforcement learning improves behaviour from evaluative feedback , 2015, Nature.

[27]  Daniel Dewey,et al.  Reinforcement Learning and the Reward Engineering Principle , 2014, AAAI Spring Symposia.

[28]  John Yearwood,et al.  On the Limitations of Scalarisation for Multi-objective Reinforcement Learning of Pareto Fronts , 2008, Australasian Conference on Artificial Intelligence.

[29]  W. D. Ross,et al.  The Right and the Good , 1930 .

[30]  Frank Dignum.  Autonomous agents and social norms , 1996 .

[31]  Ronald C. Arkin,et al.  Governing lethal behavior: Embedding ethics in a hybrid deliberative/reactive robot architecture part I: Motivation and philosophy , 2008, 2008 3rd ACM/IEEE International Conference on Human-Robot Interaction (HRI).

[32]  Peter R. Lewis,et al.  A novel adaptive weight selection algorithm for multi-objective multi-agent reinforcement learning , 2014, 2014 International Joint Conference on Neural Networks (IJCNN).

[33]  J. Fieldsend.  Multi-Objective Particle Swarm Optimisation Methods , 2004 .

[34]  Dr. Tom Murphy.  The First Level of Super Mario Bros . is Easy with Lexicographic Orderings and Time Travel , 2013 .

[35]  H. Eyre.  Introduction I , 2022 .

[36]  Aimee van Wynsberghe,et al.  Service robots, care ethics, and design , 2016, Ethics and Information Technology.

[37]  Douglas B. Lenat,et al.  EURISKO: A Program That Learns New Heuristics and Domain Concepts , 1983, Artif. Intell..

[38]  Jim Blythe,et al.  Decision-Theoretic Planning , 1999, AI Mag..

[39]  Thomas G. Dietterich,et al.  Letter to the Editor: Research Priorities for Robust and Beneficial Artificial Intelligence: An Open Letter , 2015, AI Mag..

[40]  Richard S. Sutton,et al.  Introduction to Reinforcement Learning , 1998 .

[41]  Yiannis Demiris,et al.  Evolving policies for multi-reward partially observable markov decision processes (MR-POMDPs) , 2011, GECCO '11.

[42]  Mehdi Dastani,et al.  Goal generation in the BOID architecture , 2002 .

[43]  Jessica Taylor,et al.  Quantilizers: A Safer Alternative to Maximizers for Limited Optimization , 2016, AAAI Workshop: AI, Ethics, and Society.

[44]  Carlos A. Coello Coello,et al.  Evolutionary multi-objective optimization: a historical view of the field , 2006, IEEE Comput. Intell. Mag..

[45]  Miles Brundage,et al.  Limitations and risks of machine ethics , 2014, J. Exp. Theor. Artif. Intell..

[46]  I. Kant,et al.  Grounding for the metaphysics of morals , 1981 .

[47]  Richard S. Sutton,et al.  Reinforcement Learning: An Introduction , 1998, IEEE Trans. Neural Networks.

[48]  Peter Norvig,et al.  Artificial Intelligence: A Modern Approach , 1995 .

[49]  R. Rosenfeld,et al.  Ethics , 2008 .

[50]  Patrick Lin,et al.  The Rights and Wrongs of Robot Care , 2012 .

[51]  Marcello Guarini,et al.  Particularism and the Classification and Reclassification of Moral Cases , 2006, IEEE Intelligent Systems.

[52]  David F. McAllister,et al.  An Experimental Evaluation of Software Redundancy as a Strategy For Improving Reliability , 1991, IEEE Trans. Software Eng..

[53]  Michael L. Littman,et al.  Reinforcement Learning as a Framework for Ethical Decision Making , 2016, AAAI Workshop: AI, Ethics, and Society.

[54]  P. Fishburn,et al.  Utility theory , 1980, Cognitive Choice Modeling.

[55]  Patrick Lin,et al.  Killing Made Easy: From Joysticks to Politics , 2012 .

[56]  J. Dennis,et al.  A closer look at drawbacks of minimizing weighted sums of objectives for Pareto set generation in multicriteria optimization problems , 1997 .

[57]  Andrew Critch.  Toward negotiable reinforcement learning: shifting priorities in Pareto optimal sequential decision-making , 2017, ArXiv.

[58]  Oren Etzioni,et al.  Designing AI systems that obey our laws and values , 2016, Commun. ACM.

[59]  Itamar Elhanany,et al.  On the Broad Implications of Reinforcement Learning based AGI , 2008, AGI.

[60]  Hendrik Prakken.  On how AI & Law can help autonomous systems obey the law: a position paper , 2016 .

[61]  Kyarash Shahriari,et al.  IEEE standard review — Ethically aligned design: A vision for prioritizing human wellbeing with artificial intelligence and autonomous systems , 2017, 2017 IEEE Canada International Humanitarian Technology Conference (IHTC).

[62]  Michael Anderson,et al.  An Approach to Computing Ethics , 2006, IEEE Intelligent Systems.

[63]  Michael Anderson,et al.  Machine Ethics: Creating an Ethical Intelligent Agent , 2007, AI Mag..

[64]  Pablo Noriega,et al.  Normative Multi-Agent Systems , 2013, Normative Multi-Agent Systems.

[65]  Patrick Lin,et al.  Robot Ethics: The Ethical and Social Implications of Robotics , 2011 .

[66]  Victor R. Lesser,et al.  Creating Socially Adaptive Electronic Partners: Interaction, Reasoning and Ethical Challenges , 2015, AAMAS.

[67]  Benja Fallenstein,et al.  Aligning Superintelligence with Human Interests: A Technical Research Agenda , 2015 .

[68]  Frank Dignum,et al.  Deliberative Normative Agents: Principles and Architecture , 1999, ATAL.

[69]  Daniel Dewey,et al.  Learning What to Value , 2011, AGI.

[70]  Ingemar J. Cox,et al.  Autonomous Robot Vehicles , 1990, Springer New York.

[71]  John Schulman,et al.  Concrete Problems in AI Safety , 2016, ArXiv.

[72]  Salvatore Ruggieri,et al.  A multidisciplinary survey on discrimination analysis , 2013, The Knowledge Engineering Review.

[73]  Nick Bostrom,et al.  Superintelligence: Paths, Dangers, Strategies , 2014 .

[74]  Stephen M. Omohundro,et al.  The Basic AI Drives , 2008, AGI.

[75]  J. Trinkle,et al.  Learning controllers for human-robot interaction , 2009 .