A model of pathways to artificial superintelligence catastrophe for risk and decision analysis

Abstract An artificial superintelligence (ASI) is an artificial intelligence that is significantly more intelligent than humans in all respects. Whilst ASI does not currently exist, some scholars propose that it could be created sometime in the future, and furthermore that its creation could cause a severe global catastrophe, possibly even resulting in human extinction. Given the high stakes, it is important to analyze ASI risk and factor the risk into decisions related to ASI research and development. This paper presents a graphical model of major pathways to ASI catastrophe, focusing on ASI created via recursive self-improvement. The model uses the established risk and decision analysis modelling paradigms of fault trees and influence diagrams in order to depict combinations of events and conditions that could lead to AI catastrophe, as well as intervention options that could decrease risks. The events and conditions include select aspects of the ASI itself as well as the human process of ASI research, development and management. Model structure is derived from published literature on ASI risk. The model offers a foundation for rigorous quantitative evaluation and decision-making on the long-term risk of ASI catastrophe.

[1]  N. Beckstead.  On the overwhelming importance of shaping the far future , 2013 .

[2]  Stephen M. Omohundro,et al.  The Basic AI Drives , 2008, AGI.

[3]  Richard A. Posner,et al.  Catastrophe: Risk and Response , 2004 .

[4]  Mark R. Waser.  Discovering the Foundations of a Universal System of Ethics as a Road to Safe Artificial Intelligence , 2008, AAAI Fall Symposium: Biologically Inspired Cognitive Architectures.

[5]  Nick Bostrom,et al.  Thinking Inside the Box: Controlling and Using an Oracle AI , 2012, Minds and Machines.

[6]  N. Bostrom,et al.  WHY WE NEED FRIENDLY AI , 2013, Think.

[7]  Anders Sandberg,et al.  Implications of a Software-Limited Singularity , 2013 .

[8]  I. J. Good,et al.  Speculations Concerning the First Ultraintelligent Machine , 1965, Adv. Comput..

[9]  James Babcock,et al.  Artificial General Intelligence , 2016, Lecture Notes in Computer Science.

[10]  Paul K. Davis,et al.  Studying first-strike stability with knowledge-based models of human decision-making , 1989 .

[11]  M. Ćirković.  Small Theories and Large Risks—Is Risk Analysis Relevant for Epistemology? , 2012, Risk analysis : an official publication of the Society for Risk Analysis.

[12]  R. Clemen,et al.  Soft Computing , 2002 .

[13]  Nick Bostrom,et al.  Superintelligence: Paths, Dangers, Strategies , 2014 .

[14]  Norman C. Rasmussen,et al.  The Application of Probabilistic Risk Assessment Techniques to Energy Technologies , 1981 .

[15]  Nick Bostrom,et al.  Existential Risk Prevention as Global Priority , 2013 .

[16]  Roman V Yampolskiy,et al.  Safety Engineering for Artificial General Intelligence , 2012, Topoi.

[17]  Amnon H. Eden,et al.  Singularity Hypotheses: A Scientific and Philosophical Assessment , 2013 .

[18]  Eliezer Yudkowsky,et al.  Friendly Artificial Intelligence , 2012 .

[19]  Detlof von Winterfeldt,et al.  Advances in decision analysis : from foundations to applications , 2007 .

[20]  Eliezer Yudkowsky.  Artificial Intelligence as a Positive and Negative Factor in Global Risk , 2006 .

[21]  Hiromitsu Kumamoto,et al.  Probabilistic Risk Assessment and Management for Engineers and Scientists , 1996 .

[22]  Benja Fallenstein,et al.  Aligning Superintelligence with Human Interests: A Technical Research Agenda , 2015 .

[23]  Paul Bello,et al.  Belief in The Singularity is Fideistic , 2012 .

[24]  William G. Wilkoff.  Worst-Case Scenarios , 2007 .

[25]  Miles Brundage,et al.  Limitations and risks of machine ethics , 2014, J. Exp. Theor. Artif. Intell..

[26]  J. Storrs Hall.  Machine Ethics: Ethics for Self-Improving Machines , 2011 .

[27]  Steve Omohundro,et al.  Rational Artificial Intelligence for the Greater Good , 2012 .

[28]  Jason G Matheny,et al.  Reducing the Risk of Human Extinction , 2007, Risk analysis : an official publication of the Society for Risk Analysis.

[29]  Jeff Alstott,et al.  Will We Hit a Wall? Forecasting Bottlenecks to Whole Brain Emulation Development , 2013, J. Artif. Gen. Intell..

[30]  Roman V Yampolskiy,et al.  Responses to catastrophic AGI risk: a survey , 2014 .

[31]  Steve Omohundro,et al.  Autonomous technology and the greater human good , 2014, J. Exp. Theor. Artif. Intell..

[32]  Francis S. Collins,et al.  A Framework for Decisions About Research with HPAI H5N1 Viruses , 2013, Science.

[33]  Asit P. Basu,et al.  Probabilistic Risk Analysis , 2002 .

[34]  Judea Pearl,et al.  Probabilistic reasoning in intelligent systems - networks of plausible inference , 1991, Morgan Kaufmann series in representation and reasoning.

[35]  Ben Goertzel,et al.  Nine Ways to Bias Open-Source AGI Toward Friendliness , 2012 .

[36]  Kaj Sotala,et al.  Advantages of artificial intelligences, uploads, and digital minds , 2012 .

[37]  Stuart J. Russell,et al.  Research Priorities for Robust and Beneficial Artificial Intelligence , 2015, AI Mag..

[38]  S. Baum,et al.  Analyzing and Reducing the Risks of Inadvertent Nuclear War between the United States and Russia , 2013 .