Corrigendum: Responses to catastrophic AGI risk: a survey (2015 Phys. Scr. 90 018001)

Many researchers have argued that humanity will create artificial general intelligence (AGI) within the next twenty to one hundred years. It has been suggested that AGI may inflict serious damage to human well-being on a global scale (‘catastrophic risk’). After summarizing the arguments for why AGI may pose such a risk, we review the fieldʼs proposed responses to AGI risk. We consider societal proposals, proposals for external constraints on AGI behaviors and proposals for creating AGIs that are safe due to their internal design.

[1]  David Levy,et al.  The Ethical Treatment of Artificially Conscious Robots , 2009, Int. J. Soc. Robotics.

[2]  N. Bostrom.  The Future of Human Evolution , 2001 .

[3]  Stan Franklin,et al.  Consciousness and Ethics: Artificially Conscious Moral Agents , 2011, Machine Ethics and Robot Ethics.

[4]  Carl Shulman.  Omohundro ’ s “ Basic AI Drives ” and Catastrophic Risks , 2013 .

[5]  John F. Sowa,et al.  Mapping the Landscape of Human-Level Artificial General Intelligence , 2012, AI Mag..

[6]  John P. Sullins.  When Is a Robot a Moral Agent , 2006 .

[7]  Harry G. Frankfurt,et al.  The importance of what we care about: Freedom of the will and the concept of a person , 1971 .

[8]  Kaj Sotala,et al.  Advantages of artificial intelligences, uploads, and digital minds , 2012 .

[9]  Diana F. Spears,et al.  Assuring the Behavior of Adaptive Agents , 2006 .

[10]  Colin Allen,et al.  Prolegomena to any future artificial moral agent , 2000, J. Exp. Theor. Artif. Intell..

[11]  Mark Walker,et al.  Human Extinction and Farsighted Universal Surveillance , 2012, Int. J. Technoethics.

[12]  Pei Wang,et al.  Motivation Management in AGI Systems , 2012, AGI.

[13]  Nick Bostrom,et al.  Superintelligence: Paths, Dangers, Strategies , 2014 .

[14]  Edward W. Felten,et al.  Timing attacks on Web privacy , 2000, CCS.

[15]  R. Penrose,et al.  How Long Until Human-Level AI ? Results from an Expert Assessment , 2011 .

[16]  Thomas Robbins,et al.  Conversion and “Brainwashing” in New Religious Movements , 2008 .

[17]  Javier Snaider,et al.  The LIDA Framework as a General Tool for AGI , 2011, AGI.

[18]  Julian Savulescu,et al.  The perils of cognitive enhancement and the urgent imperative to enhance the moral character of humanity , 2008 .

[19]  Ben Goertzel,et al.  Why an Intelligence Explosion is Probable , 2012 .

[20]  George J Annas,et al.  Protecting the endangered human: toward an international treaty prohibiting cloning and inheritable alterations. , 2002, American journal of law & medicine.

[21]  Steve Omohundro,et al.  Rational Artificial Intelligence for the Greater Good , 2012 .

[22]  Bill Hibbard,et al.  Avoiding Unintended AI Behaviors , 2012, AGI.

[23]  Raymond C. Kurzweil,et al.  The Singularity Is Near , 2018, The Infinite Desire for Growth.

[24]  Ben Goertzel,et al.  Nine Ways to Bias Open-Source AGI Toward Friendliness , 2012 .

[25]  A. Tversky,et al.  The framing of decisions and the psychology of choice. , 1981, Science.

[26]  Robert Axelrod,et al.  The Evolution of Strategies in the Iterated Prisoner's Dilemma , 2001 .

[27]  K. Warwick.  Cyborg morals, cyborg values, cyborg ethics , 2003, Ethics and Information Technology.

[28]  Eliezer Yudkowsky,et al.  The Ethics of Artificial Intelligence , 2014, Artificial Intelligence Safety and Security.

[29]  Rodney Brooks,et al.  I, Rodney Brooks, am a robot , 2008, IEEE Spectrum.

[30]  Michael Anderson,et al.  An Approach to Computing Ethics , 2006, IEEE Intelligent Systems.

[31]  Butler W. Lampson,et al.  A note on the confinement problem , 1973, CACM.

[32]  Thomas D. Nielsen,et al.  Learning a decision maker's utility function from (possibly) inconsistent behavior , 2004, Artif. Intell..

[33]  D. Kipnis,et al.  Does power corrupt? , 1972, Journal of personality and social psychology.

[34]  Gabriel Hallevy.  The Criminal Liability of Artificial Intelligence Entities , 2010 .

[35]  Jürgen Schmidhuber,et al.  A Family of Gödel Machine Implementations , 2011, AGI.

[36]  Andreas Terzis,et al.  My Botnet Is Bigger Than Yours (Maybe, Better Than Yours): Why Size Estimates Remain Challenging , 2007, HotBots.

[37]  Bill Hibbard,et al.  Decision Support for Safe AI Design , 2012, AGI.

[38]  Batya Friedman,et al.  Human agency and responsible computing: Implications for computer system design , 1992, J. Syst. Softw..

[39]  Bruce M. McLaren,et al.  Extensionally defining principles and cases in ethics: An AI model , 2003, Artif. Intell..

[40]  Kenneth J. Hayworth,et al.  ELECTRON IMAGING TECHNOLOGY FOR WHOLE BRAIN NEURAL CIRCUIT MAPPING , 2012 .

[41]  M McLarenBruce.  Computational Models of Ethical Reasoning , 2006 .

[42]  Roman V. Yampolskiy,et al.  Leakproofing the Singularity Artificial Intelligence Confinement Problem , 2012 .

[43]  Bill Hibbard,et al.  Super-intelligent machines , 2012, COMG.

[44]  Daniel Dewey,et al.  Learning What to Value , 2011, AGI.

[45]  Carl Shulman,et al.  Machine Ethics and Superintelligence , 2009 .

[46]  Amnon H. Eden,et al.  Singularity Hypotheses: A Scientific and Philosophical Assessment , 2013 .

[47]  K. Warwick.  Implications and consequences of robots with biological brains , 2010, Ethics and Information Technology.

[48]  Carl Shulman.  The Singularity Institute Whole Brain Emulation and the Evolution of Superorganisms , 2012 .

[49]  Carl Shulman,et al.  Which Consequentialism ? Machine Ethics and Moral Divergence , 2013 .

[50]  Y. Trope,et al.  Construal-level theory of psychological distance. , 2010, Psychological review.

[51]  Vitaly Shmatikov,et al.  De-anonymizing Social Networks , 2009, 2009 30th IEEE Symposium on Security and Privacy.

[52]  Roman V Yampolskiy,et al.  Safety Engineering for Artificial General Intelligence , 2012, Topoi.

[53]  Seth D. Baum,et al.  The great downside dilemma for risky emerging technologies , 2014 .

[54]  Selmer Bringsjord,et al.  On How to Build a Moral Machine , 2013 .

[55]  Richard B. Brandt.  A theory of the good and the right , 1979 .

[56]  Paul Bello,et al.  Belief in The Singularity is Fideistic , 2012 .

[57]  Christopher Grau.  There is no ‘ I ’ in ‘ Robot ’ : Robotic Utilitarians and Utilitarian Robots , 2005 .

[58]  Marcello Guarini,et al.  Particularism and the Classification and Reclassification of Moral Cases , 2006, IEEE Intelligent Systems.

[59]  Ronald C. Arkin,et al.  Governing Lethal Behavior in Autonomous Robots , 2009 .

[60]  R. Hanson.  Economic Growth Given Machine Intelligence , 2000 .

[61]  Thomas M. Powers,et al.  Incremental Machine Ethics , 2011, IEEE Robotics & Automation Magazine.

[62]  Marcus Hutter,et al.  Can Intelligence Explode? , 2012, ArXiv.

[63]  Eliezer Yudkowsky.  Complex Value Systems are Required to Realize Valuable Futures , 2011 .

[64]  J. Gips.  Towards the ethical robot , 1995 .

[65]  Mark R. Waser.  Discovering the Foundations of a Universal System of Ethics as a Road to Safe Artificial Intelligence , 2008, AAAI Fall Symposium: Biologically Inspired Cognitive Architectures.

[66]  Kevin D. Ashley,et al.  Reasoning with Reasons in Case-Based Comparisons , 1995, ICCBR.

[67]  T. Gelder,et al.  What Might Cognition Be, If Not Computation? , 1995 .

[68]  J. Tenenbaum,et al.  Theory-based Bayesian models of inductive learning and reasoning , 2006, Trends in Cognitive Sciences.

[69]  Philippe Golle,et al.  On the Anonymity of Home/Work Location Pairs , 2009, Pervasive.

[70]  The Singularity and the State of the Art in Artificial Intelligence , 2014 .

[71]  Ben Goertzel,et al.  WHEN SHOULD TWO MINDS BE CONSIDERED VERSIONS OF ONE ANOTHER , 2012 .

[72]  Robin Hanson,et al.  Shall We Vote on Values, But Bet on Beliefs? , 2013 .

[73]  DegabrieleJean Paul,et al.  Provable Security in the Real World , 2011, S&P 2011.

[74]  Jürgen Schmidhuber,et al.  Ultimate Cognition à la Gödel , 2009, Cognitive Computation.

[75]  Michael Smith,et al.  DESIRES, VALUES, REASONS, AND THE DUALISM OF PRACTICAL. REASON , 2009 .

[76]  Bill Hibbard,et al.  Model-based Utility Functions , 2011, J. Artif. Gen. Intell..

[77]  Oren Etzioni,et al.  The First Law of Robotics (A Call to Arms) , 1994, AAAI.

[78]  Stan Franklin,et al.  A Conceptual and Computational Model of Moral Decision Making in Human and Artificial Agents , 2010, Top. Cogn. Sci..

[79]  Chuen-Tsai Sun,et al.  Toward the Human–Robot Co-Existence Society: On Safety Intelligence for Next Generation Robots , 2009, Int. J. Soc. Robotics.

[80]  Carl Shulman,et al.  Superintelligence Does Not Imply Benevolence , 2013 .

[81]  Philippe Verdoux,et al.  Emerging Technologies and the Future of Philosophy , 2011 .

[82]  S. Bamford,et al.  A FRAMEWORK FOR APPROACHES TO TRANSFER OF A MIND'S SUBSTRATE , 2012 .

[83]  David Lewis,et al.  Dispositional Theories of Value , 1989 .

[84]  Shane Legg,et al.  A Collection of Definitions of Intelligence , 2007, AGI.

[85]  Vern Paxson,et al.  How to Own the Internet in Your Spare Time , 2002, USENIX Security Symposium.

[86]  Eric Dietrich,et al.  After the Humans are Gone , 2007 .

[87]  Randal A. Koene,et al.  EXPERIMENTAL RESEARCH IN WHOLE BRAIN EMULATION: THE NEED FOR INNOVATIVE IN VIVO MEASUREMENT TECHNIQUES , 2012 .

[88]  Jonathan Haidt,et al.  The Happiness Hypothesis , 2006 .

[89]  R. Hanson.  Economics of the singularity , 2008, IEEE Spectrum.

[90]  Peter Dayan,et al.  Models of value and choice. , 2012 .

[91]  Vitaly Shmatikov,et al.  Robust De-anonymization of Large Sparse Datasets , 2008, 2008 IEEE Symposium on Security and Privacy (sp 2008).

[92]  David Brin,et al.  The Transparent Society , 1998 .

[93]  Thomas M. Powers.  Prospects for a Kantian Machine , 2006, IEEE Intelligent Systems.

[94]  Ben Goertzel,et al.  Artificial General Intelligence , 2017, Lecture Notes in Computer Science.

[95]  Patrick Lin,et al.  Moral Machines: Contradiction in Terms or Abdication of Human Responsibility? , 2012 .

[96]  Wendell Wallach,et al.  Why Machine Ethics? , 2006, IEEE Intelligent Systems.

[97]  John P. Sullins.  Ethics and Artificial life: From Modeling to Moral Agents , 2005, Ethics and Information Technology.

[98]  G. Amdhal,et al.  Validity of the single processor approach to achieving large scale computing capabilities , 1967, AFIPS '67 (Spring).

[99]  Sarah K. Murnen,et al.  The effect of experimental presentation of thin media images on body satisfaction: a meta-analytic review. , 2002, The International journal of eating disorders.

[100]  Stephen Omohundro,et al.  The Nature of Self-Improving Artificial Intelligence , 2008 .

[101]  R D Hare,et al.  Psychopathy and the predictive validity of the PCL-R: an international perspective. , 2000, Behavioral sciences & the law.

[102]  Irving John Good,et al.  Some future social repercussions of computers , 1970 .

[103]  David Zimmerman.  Why Richard Brandt Does Not Need Cognitive Psychotherapy, and Other Glad News about Idealized Preference Theories in Meta-Ethics , 2003 .

[104]  Francis Heylighen,et al.  A brain in a vat cannot break out: why the singularity must be extended, embedded and embodied , 2012 .

[105]  Gregory Clark,et al.  A farewell to alms , 2005, BMJ : British Medical Journal.

[106]  Mark Walker,et al.  Personal Identity and Uploading , 2011 .

[107]  L. Versenyi,et al.  Can Robots be Moral? , 1974, Ethics.

[108]  R. J. Solomon Off,et al.  The time scale of artificial intelligence: Reflections on social effects , 1985 .

[109]  W. S. McCulloch,et al.  Toward some circuitry of ethical robots or an observational science of the genesis of social evaluation in the mind-like behavior of artifacts , 1956 .

[110]  P. Hopkins,et al.  WHY UPLOADING WILL NOT WORK, OR, THE GHOSTS HAUNTING TRANSHUMANISM , 2012 .

[111]  Joy Bill,et al.  Why the future doesn’t need us , 2003 .

[112]  Philippe Verdoux,et al.  Risk Mysterianism and Cognitive Boosters , 2010 .

[113]  Wendell Wallach,et al.  Robot minds and human ethics: the need for a comprehensive model of moral decision making , 2010, Ethics and Information Technology.

[114]  Hans P. Moravec.  When will computer hardware match the human brain , 1998 .

[115]  Blay Whitby,et al.  Reflections on artificial intelligence , 1996 .

[116]  Peter Eckersley,et al.  Is Brain Emulation Dangerous? , 2013, J. Artif. Gen. Intell..

[117]  Peter D. Turney.  Controlling Super-Intelligent Machines , 1991 .

[118]  Catrin Finkenauer,et al.  Breaking the Rules to Rise to Power , 2011 .

[119]  W. Napier,et al.  Hazards from comets and asteroids , 2008 .

[120]  Olle Häggström.  Emerging technologies and the future of humanity , 2014 .

[121]  Laurent Orseau,et al.  Self-Modification and Mortality in Artificial Agents , 2011, AGI.

[122]  Luke Muehlhauser,et al.  The Singularity and Machine Ethics , 2012 .

[123]  Stacey Tantleff-Dunn,et al.  THE IMPACT OF MEDIA EXPOSURE ON MALES BODY IMAGE , 2004 .

[124]  William Daley,et al.  Mitigating potential hazards to humans from the development of intelligent machines , 2011 .

[125]  Eliezer Yukdowsky.  Artificial Intelligence as a Positive and Negative Factor in Global Risk , 2008 .

[126]  Mark R. Waser.  A Safe Ethical System for Intelligent Machines , 2009, AAAI Fall Symposium: Biologically Inspired Cognitive Architectures.

[127]  Wendell Wallach,et al.  Framing robot arms control , 2012, Ethics and Information Technology.

[128]  David Benatar,et al.  Better Never to Have Been , 2006 .

[129]  Anders Sandberg,et al.  An Overview of Models of Technological Singularity , 2013 .

[130]  Ben Goertzel,et al.  Stages of Ethical Development in Artificial General Intelligence Systems , 2008, AGI.

[131]  L Sweeney,et al.  Weaving Technology and Policy Together to Maintain Confidentiality , 1997, Journal of Law, Medicine & Ethics.

[132]  I. J. Good,et al.  Speculations Concerning the First Ultraintelligent Machine , 1965, Adv. Comput..

[133]  Ravi Prakash,et al.  The conscious access hypothesis: Explaining the consciousness , 2008, Indian journal of psychiatry.

[134]  P. Railton.  Facts and Values , 1986 .

[135]  N Wiener,et al.  Some moral and technical consequences of automation , 1960, Science.

[136]  Ben Goertzel,et al.  OpenCog: A Software Framework for Integrative Artificial General Intelligence , 2008, AGI.

[137]  L. McCauley.  AI Armageddon and the Three Laws of Robotics , 2007, Ethics and Information Technology.

[138]  Laurent Orseau,et al.  Delusion, Survival, and Intelligent Agents , 2011, AGI.

[139]  Edmund T. Rolls,et al.  Introduction to Connectionist Modelling of Cognitive Processes , 1998 .

[140]  David B Pisoni,et al.  Cochlear implants and spoken language processing abilities: review and assessment of the literature. , 2010, Restorative neurology and neuroscience.