Responses to Catastrophic AGI Risk : A Survey Kaj Sotala Machine Intelligence Research Institute

Many researchers have argued that humanity will create artificial general intelligence (AGI) within the next twenty to one hundred years. It has been suggested that AGI may pose a catastrophic risk to humanity. After summarizing the arguments for why AGI may pose such a risk, we survey the field’s proposed responses to AGI risk. We consider societal proposals, proposals for external constraints on AGI behaviors, and proposals for creating AGIs that are safe due to their internal design. Sotala, Kaj and Roman V. Yampolskiy. 2013. Responses to Catastrophic AGI Risk: A Survey. Technical Report, 2013-2. Machine Intelligence Research Institute, Berkeley, CA. Last modified June 28, 2013.

[1]  R. Kurzweil,et al.  The Singularity Is Near: When Humans Transcend Biology , 2006 .

[2]  Catrin Finkenauer,et al.  Breaking the Rules to Rise to Power , 2011 .

[3]  Ronald C. Arkin,et al.  Governing Lethal Behavior in Autonomous Robots , 2009 .

[4]  Ben Goertzel,et al.  Stages of Ethical Development in Artificial General Intelligence Systems , 2008, AGI.

[5]  J. Storrs Hall.  Beyond AI: Creating the Conscience of the Machine , 2007 .

[6]  Jürgen Schmidhuber,et al.  A Family of Gödel Machine Implementations , 2011, AGI.

[7]  Christopher Grau,et al.  There Is No "I" in "Robot": Robots and Utilitarianism , 2006, IEEE Intelligent Systems.

[8]  G. Amdhal,et al.  Validity of the single processor approach to achieving large scale computing capabilities , 1967, AFIPS '67 (Spring).

[9]  Butler W. Lampson,et al.  A note on the confinement problem , 1973, CACM.

[10]  F. Salzano.  Primates and philosophers: how morality evolved , 2007 .

[11]  “Other Worlds than Ours” , 1870, Nature.

[12]  Kaj Sotala,et al.  Advantages of artificial intelligences, uploads, and digital minds , 2012 .

[13]  D. Dennis,et al.  Seeing Like a State: How Certain Schemes to Improve the Human Condition Have Failed , 1998 .

[14]  Chuen-Tsai Sun,et al.  Safety Intelligence and Legal Machine Language: Do We Need the Three Laws of Robotics? , 2008 .

[15]  Roman V. Yampolskiy,et al.  Leakproofing the Singularity Artificial Intelligence Confinement Problem , 2012 .

[16]  Joan R. Rosés Vendoiro,et al.  A Farewell to Alms: A Brief Economic History of the World , 2007 .

[17]  John P. Sullins.  Ethics and Artificial life: From Modeling to Moral Agents , 2005, Ethics and Information Technology.

[18]  Kenneth G. Paterson,et al.  Provable Security in the Real World , 2011, IEEE Security & Privacy.

[19]  Stephen Omohundro,et al.  The Nature of Self-Improving Artificial Intelligence , 2008 .

[20]  John F. Sowa,et al.  Mapping the Landscape of Human-Level Artificial General Intelligence , 2012, AI Mag..

[21]  John P. Sullins.  When Is a Robot a Moral Agent , 2006 .

[22]  Michael Anderson,et al.  An Approach to Computing Ethics , 2006, IEEE Intelligent Systems.

[23]  Roger Clarke,et al.  Asimov's Laws of Robotics: Implications for Information Technology - Part 2 , 1993, Computer.

[24]  Shane Legg,et al.  A Collection of Definitions of Intelligence , 2007, AGI.

[25]  Wendell Wallach,et al.  Why Machine Ethics? , 2006, IEEE Intelligent Systems.

[26]  Y. Trope,et al.  Construal-level theory of psychological distance. , 2010, Psychological review.

[27]  Andreas Terzis,et al.  My Botnet Is Bigger Than Yours (Maybe, Better Than Yours): Why Size Estimates Remain Challenging , 2007, HotBots.

[28]  Laurent Orseau,et al.  Self-Modification and Mortality in Artificial Agents , 2011, AGI.

[29]  J. Bach,et al.  Artificial General Intelligence: 5th International Conference, AGI 2012, Oxford, UK, December 8-11, 2012. Proceedings , 2012 .

[30]  A. Nordmann.  If and Then: A Critique of Speculative NanoEthics , 2007, The Ethics of Nanotechnology, Geoengineering and Clean Energy.

[31]  Philippe Verdoux,et al.  Emerging Technologies and the Future of Philosophy , 2011 .

[32]  Diana F. Spears.  Asimov's Laws: Current Progress , 2002, FAABS.

[33]  The Singularity and the State of the Art in Artificial Intelligence , 2014 .

[34]  John Othick,et al.  The Rise and Decline of Nations: Economic Growth, Stagflation, and Social Rigidities. , 1983 .

[35]  Dorothy Mitchell.  A Theory of the Good and the Right , 1980 .

[36]  Nick Bostrom,et al.  The Superintelligent Will: Motivation and Instrumental Rationality in Advanced Artificial Agents , 2012, Minds and Machines.

[37]  David Sobel,et al.  Full Information Accounts of Well-Being , 1994, Ethics.

[38]  G. Moskowitz,et al.  The Implicit Volition Model: On the Preconscious Regulation of Temporarily Adopted Goals , 2004 .

[39]  B. Baars.  Global workspace theory of consciousness: toward a cognitive neuroscience of human experience. , 2005, Progress in brain research.

[40]  P. Asaro.  Robots and Responsibility from a Legal Perspective , 2007 .

[41]  Wendell Wallach,et al.  Robot minds and human ethics: the need for a comprehensive model of moral decision making , 2010, Ethics and Information Technology.

[42]  C. Cloos,et al.  The Utilibot Project : An Autonomous Mobile Robot Based on Utilitarianism , 2005 .

[43]  Batya Friedman,et al.  Human agency and responsible computing: Implications for computer system design , 1992, J. Syst. Softw..

[44]  D. Chalmers.  The conscious mind: in search of a fundamental theory , 1996 .

[45]  Zachary J. McDowell.  The machine question: Critical perspectives on AI, robots, and ethics , 2014, New Media Soc..

[46]  Michael Anderson,et al.  MedEthEx: Toward a Medical Ethics Advisor , 2005, AAAI Fall Symposium: Caring Machines.

[47]  David C. Plaut,et al.  Connectionist Modeling of Language: Examples and Implications , 2000 .

[48]  David Levy,et al.  The Ethical Treatment of Artificially Conscious Robots , 2009, Int. J. Soc. Robotics.

[49]  G. Harris,et al.  Treatment of Psychopathy: A Review of Empirical Findings. , 2006 .

[50]  M. Hauskeller.  MY BRAIN, MY MIND, AND I: SOME PHILOSOPHICAL ASSUMPTIONS OF MIND-UPLOADING , 2012 .

[51]  N. Bostrom.  Are We Living in a Computer Simulation , 2003 .

[52]  Ben Goertzel,et al.  WHEN SHOULD TWO MINDS BE CONSIDERED VERSIONS OF ONE ANOTHER , 2012 .

[53]  Jürgen Schmidhuber,et al.  Ultimate Cognition à la Gödel , 2009, Cognitive Computation.

[54]  R. J. Solomon Off,et al.  The time scale of artificial intelligence: Reflections on social effects , 1985 .

[55]  K. Eric Drexler,et al.  Engines of Creation , 1986 .

[56]  Robin R. Murphy,et al.  Beyond Asimov: The Three Laws of Responsible Robotics , 2009, IEEE Intelligent Systems.

[57]  Howard Barnum,et al.  The Beginning of Infinity: Explanations That Transform the World , 2012 .

[58]  Eliezer Yudkowsky.  Complex Value Systems are Required to Realize Valuable Futures , 2011 .

[59]  J. Bryson.  Robots should be slaves , 2010 .

[60]  David Zimmerman.  Why Richard Brandt Does Not Need Cognitive Psychotherapy, and Other Glad News about Idealized Preference Theories in Meta-Ethics , 2003 .

[61]  B. Baars,et al.  LIDA: A Working Model of Cognition , 2006 .

[62]  David B Pisoni,et al.  Cochlear implants and spoken language processing abilities: review and assessment of the literature. , 2010, Restorative neurology and neuroscience.

[63]  N. Bostrom.  Technological revolutions: ethics and policy in the dark , 2007 .

[64]  Philippe Verdoux,et al.  Risk Mysterianism and Cognitive Boosters , 2010 .

[65]  Miri Visiting Fellow,et al.  Coherent Extrapolated Volition: A Meta-Level Approach to Machine Ethics , 2010 .

[66]  C. Allen,et al.  Moral Machines: Teaching Robots Right from Wrong , 2008 .

[67]  Patrick Lin,et al.  Robot Ethics: The Ethical and Social Implications of Robotics , 2011 .

[68]  Stan Franklin,et al.  THE LIDA ARCHITECTURE: ADDING NEW MODES OF LEARNING TO AN INTELLIGENT, AUTONOMOUS, SOFTWARE AGENT , 2006 .

[69]  Thomas Douglas.  Moral Enhancement , 2008, Journal of applied philosophy.

[70]  Susan Leigh Anderson,et al.  Machine Ethics: The Unacceptability of Asimov's Three Laws of Robotics as a Basis for Machine Ethics , 2011 .

[71]  W. S. McCulloch,et al.  Toward some circuitry of ethical robots or an observational science of the genesis of social evaluation in the mind-like behavior of artifacts , 1956 .

[72]  Vernor Vinge,et al.  ==================================================================== the Coming Technological Singularity: How to Survive in the Post-human Era , 2022 .

[73]  Kenneth J. Hayworth,et al.  ELECTRON IMAGING TECHNOLOGY FOR WHOLE BRAIN NEURAL CIRCUIT MAPPING , 2012 .

[74]  H. Mounce,et al.  Reason and Morality , 1986 .

[75]  D. Mueller.  Public Choice III: What have we learned? , 1979 .

[76]  B. Baars.  The conscious access hypothesis: origins and recent evidence , 2002, Trends in Cognitive Sciences.

[77]  David Moore,et al.  Code-Red: a case study on the spread and victims of an internet worm , 2002, IMW '02.

[78]  Edmund T. Rolls,et al.  Introduction to Connectionist Modelling of Cognitive Processes , 1998 .

[79]  Alexey Potapov,et al.  Universal empathy and ethical bias for artificial general intelligence , 2013, J. Exp. Theor. Artif. Intell..

[80]  Z. Pylyshyn.  Robot's Dilemma: The Frame Problem in Artificial Intelligence , 1987 .

[81]  Aaron Sloman,et al.  The St. Thomas Common Sense Symposium: Designing Architectures for Human-Level Intelligence , 2004, AI Mag..

[82]  Paul Bello,et al.  Belief in The Singularity is Fideistic , 2012 .

[83]  Attila Tanyi.  An Essay on the Desire-Based Reasons Model , 2006 .

[84]  Oren Etzioni,et al.  The First Law of Robotics (A Call to Arms) , 1994, AAAI.

[85]  Edward W. Felten,et al.  Bubble Trouble: Off-Line De-Anonymization of Bubble Forms , 2011, USENIX Security Symposium.

[86]  Erik T. Mueller,et al.  Achieving Human-Level Intelligence through Integrated Systems and Research: Introduction to This Special Issue , 2006, AI Mag..

[87]  Dawn Xiaodong Song,et al.  On the Feasibility of Internet-Scale Author Identification , 2012, 2012 IEEE Symposium on Security and Privacy.

[88]  Randal A. Koene,et al.  EXPERIMENTAL RESEARCH IN WHOLE BRAIN EMULATION: THE NEED FOR INNOVATIVE IN VIVO MEASUREMENT TECHNIQUES , 2012 .

[89]  Carl Shulman,et al.  Superintelligence Does Not Imply Benevolence , 2013 .

[90]  T. Gelder,et al.  What Might Cognition Be, If Not Computation? , 1995 .

[91]  N. Bostrom,et al.  Global Catastrophic Risks , 2008 .

[92]  S. Bamford,et al.  A FRAMEWORK FOR APPROACHES TO TRANSFER OF A MIND'S SUBSTRATE , 2012 .

[93]  Roman V Yampolskiy,et al.  Safety Engineering for Artificial General Intelligence , 2012, Topoi.

[94]  Thomas M. Powers.  Prospects for a Kantian Machine , 2006, IEEE Intelligent Systems.

[95]  Colin Allen,et al.  Prolegomena to any future artificial moral agent , 2000, J. Exp. Theor. Artif. Intell..

[96]  Marcus Hutter,et al.  Can Intelligence Explode? , 2012, ArXiv.

[97]  Ben Goertzel,et al.  OpenCog: A Software Framework for Integrative Artificial General Intelligence , 2008, AGI.

[98]  R D Hare,et al.  Psychopathy and the predictive validity of the PCL-R: an international perspective. , 2000, Behavioral sciences & the law.

[99]  Javier Snaider,et al.  The LIDA Framework as a General Tool for AGI , 2011, AGI.

[100]  Lawrence B. Solum.  Legal Personhood for Artificial Intelligences , 2008, Machine Ethics and Robot Ethics.

[101]  P. Hopkins,et al.  WHY UPLOADING WILL NOT WORK, OR, THE GHOSTS HAUNTING TRANSHUMANISM , 2012 .

[102]  Thomas D. Nielsen,et al.  Learning a decision maker's utility function from (possibly) inconsistent behavior , 2004, Artif. Intell..

[103]  P. Hayes,et al.  Cognitive Wheels : The Frame Problem of AI , 2022 .

[104]  Bruce M. McLaren,et al.  Computational Models of Ethical Reasoning: Challenges, Initial Steps, and Future Directions , 2006, IEEE Intelligent Systems.

[105]  Ben Goertzel,et al.  CogPrime : An Integrative Architecture for Embodied Artificial General Intelligence , 2012 .

[106]  Laurent Orseau,et al.  Delusion, Survival, and Intelligent Agents , 2011, AGI.

[107]  Peter Suber.  Saving Machines From Themselves: The Ethics of Deep Self-Modification , 2001 .

[108]  Ben Goertzel,et al.  Nine Ways to Bias Open-Source AGI Toward Friendliness , 2012 .

[109]  Erik Brynjolfsson,et al.  Race against the machine : how the digital revolution is accelerating innovation, driving productivity, and irreversibly transforming employment and the economy , 2011 .

[110]  Eric Dietrich,et al.  After the Humans are Gone , 2007 .

[111]  T. Wilson.  Strangers to Ourselves: Discovering the Adaptive Unconscious , 2002 .

[112]  Mark Walker,et al.  Human Extinction and Farsighted Universal Surveillance , 2012, Int. J. Technoethics.

[113]  F. Gregory.  The conscious mind: In search of a fundamental theory , 1998 .

[114]  Carl Shulman,et al.  Arms Control and Intelligence Explosions , 2013 .

[115]  Mark Walker,et al.  Personal Identity and Uploading , 2011 .

[116]  Anthony F. Beavers.  Moral Machines and the Threat of Ethical Nihilism , 2010 .

[117]  Eliezer Yudkowsky.  Artificial Intelligence as a Positive and Negative Factor in Global Risk , 2006 .

[118]  Amnon H. Eden,et al.  Singularity Hypotheses: A Scientific and Philosophical Assessment , 2013 .

[119]  Thomas Robbins,et al.  Conversion and “Brainwashing” in New Religious Movements , 2008 .

[120]  Relinquishment or Regulation: Dealing with Apocalyptic Technological Threats , 2001 .

[121]  Carl Shulman,et al.  Machine Ethics and Superintelligence , 2009 .

[122]  John R. Searle,et al.  The Rediscovery of the Mind , 1995, Artif. Intell..

[123]  Randal A. Koene,et al.  Embracing Competitive Balance: The Case for Substrate-Independent Minds and Whole Brain Emulation , 2012 .

[124]  A. Nordmann.  Singular simplicity , 2008, IEEE Spectrum.

[125]  Mark R. Waser.  Discovering the Foundations of a Universal System of Ethics as a Road to Safe Artificial Intelligence , 2008, AAAI Fall Symposium: Biologically Inspired Cognitive Architectures.

[126]  Melissa J. Ferguson,et al.  Implicit motivation: Past, present, and future. , 2008 .

[127]  W. Napier,et al.  Hazards from comets and asteroids , 2008 .

[128]  Francis Heylighen,et al.  Accelerating Socio-Technological Evolution: from ephemeralization and stigmergy to the global brain , 2007, ArXiv.

[129]  Christopher A. Rouff,et al.  Agent Technology from a Formal Perspective , 2006 .

[130]  Nick Bostrom,et al.  Thinking Inside the Box: Controlling and Using an Oracle AI , 2012, Minds and Machines.

[131]  Nicholas Agar,et al.  A Peer-reviewed Electronic Journal Published by the Institute for Ethics and Emerging Technologies Ray Kurzweil and Uploading: Just Say No! , 2011 .

[132]  Mark R. Waser,et al.  Rational Universal Benevolence: Simpler, Safer, and Wiser Than "Friendly AI" , 2011, AGI.

[133]  B. Weatherson.  Should We Respond to Evil with Indifference , 2005 .

[134]  S. Pinker.  The Blank Slate: The Modern Denial of Human Nature , 2002 .

[135]  Vitaly Shmatikov,et al.  Robust De-anonymization of Large Sparse Datasets , 2008, 2008 IEEE Symposium on Security and Privacy (sp 2008).

[136]  James L. McClelland,et al.  Connectionist models of cognition. , 2008 .

[137]  A. Tversky,et al.  The framing of decisions and the psychology of choice. , 1981, Science.

[138]  Stacey Tantleff-Dunn,et al.  THE IMPACT OF MEDIA EXPOSURE ON MALES BODY IMAGE , 2004 .

[139]  Steve Omohundro,et al.  Rational Artificial Intelligence for the Greater Good , 2012 .

[140]  Peter Eckersley,et al.  Is Brain Emulation Dangerous? , 2013, J. Artif. Gen. Intell..

[141]  Kevin D. Ashley,et al.  Reasoning with Reasons in Case-Based Comparisons , 1995, ICCBR.

[142]  D. Chalmers.  The Singularity: a Philosophical Analysis , 2010 .

[143]  Blay Whitby.  Reflections on Artificial Intelligence: The Legal, Moral and Ethical Dimensions , 1996 .

[144]  Stan Franklin,et al.  Consciousness and Ethics: Artificially Conscious Moral Agents , 2011, Machine Ethics and Robot Ethics.

[145]  Bruce M. McLaren,et al.  Extensionally defining principles and cases in ethics: An AI model , 2003, Artif. Intell..

[146]  Stan Franklin,et al.  A Conceptual and Computational Model of Moral Decision Making in Human and Artificial Agents , 2010, Top. Cogn. Sci..

[147]  Julian Savulescu,et al.  The perils of cognitive enhancement and the urgent imperative to enhance the moral character of humanity , 2008 .

[148]  Harry G. Frankfurt,et al.  The importance of what we care about: Freedom of the will and the concept of a person , 1971 .

[149]  A. Jenkins.  Artificial intelligence and the real world , 2003 .

[150]  Luke Muehlhauser,et al.  Intelligence Explosion: Evidence and Import , 2012 .

[151]  Michael Anderson,et al.  Machine Ethics , 2011 .

[152]  Thomas M. Powers,et al.  Incremental Machine Ethics , 2011, IEEE Robotics & Automation Magazine.

[153]  Pei Wang,et al.  Motivation Management in AGI Systems , 2012, AGI.

[154]  L. Damm,et al.  Moral Machines: Teaching Robots Right from Wrong , 2012 .

[155]  Bonnie Docherty,et al.  Losing Humanity : The Case Against Killer Robots , 2012 .

[156]  Bill Hibbard,et al.  Super-intelligent machines , 2012, COMG.

[157]  Vern Paxson,et al.  How to Own the Internet in Your Spare Time , 2002, USENIX Security Symposium.

[158]  C. Allen,et al.  Artificial Morality: Top-down, Bottom-up, and Hybrid Approaches , 2005, Ethics and Information Technology.

[159]  George J Annas,et al.  Protecting the endangered human: toward an international treaty prohibiting cloning and inheritable alterations. , 2002, American journal of law & medicine.

[160]  Hugo de Garis,et al.  The Artilect War: Cosmists vs. Terrans. A Bitter Controversy Concerning Whether Humanity Should Build Godlike Massively Intelligent Machines , 2008, AGI.

[161]  A. Elga.  Defeating Dr. Evil with Self-Locating Belief , 2004 .

[162]  I. J. Good,et al.  Speculations Concerning the First Ultraintelligent Machine , 1965, Adv. Comput..

[163]  P. Railton.  Facts and Values , 1986 .

[164]  In the mind of the machine: The breakthrough in artificial intelligence , 1998 .

[165]  Nancy M. Amato,et al.  A Roadmap for US Robotics - From Internet to Robotics 2020 Edition , 2021, Found. Trends Robotics.

[166]  D. Brin.  The Transparent Society: Will Technology Force Us to Choose Between Privacy and Freedom? , 1998 .

[167]  Liz Stillwaggon Swan,et al.  DIGITAL IMMORTALITY: SELF OR 0010110? , 2012 .

[168]  L Sweeney,et al.  Weaving Technology and Policy Together to Maintain Confidentiality , 1997, Journal of Law, Medicine & Ethics.

[169]  Michael Smith,et al.  DESIRES, VALUES, REASONS, AND THE DUALISM OF PRACTICAL. REASON , 2009 .

[170]  Chris Armen,et al.  Towards Machine Ethics: Implementing Two Action-Based Ethical Theories , 2005 .

[171]  D. Kipnis,et al.  Does power corrupt? , 1972, Journal of personality and social psychology.

[172]  Hans P. Moravec.  Robot: Mere Machine to Transcendent Mind , 1998 .

[173]  Milind Tambe,et al.  Revisiting Asimov's First Law: A Response to the Call to Arms , 2001, ATAL.

[174]  Jürgen Schmidhuber,et al.  Artificial General Intelligence - 4th International Conference, AGI 2011, Mountain View, CA, USA, August 3-6, 2011. Proceedings , 2011, AGI.

[175]  Siobhan Campbell,et al.  Darwin among the machines , 2009 .

[176]  Marcello Guarini,et al.  Particularism and the Classification and Reclassification of Moral Cases , 2006, IEEE Intelligent Systems.

[177]  Stuart Armstrong,et al.  How We're Predicting AI - or Failing to , 2015 .

[178]  Irving John Good,et al.  Some future social repercussions of computers , 1970 .

[179]  R. Penrose,et al.  How Long Until Human-Level AI ? Results from an Expert Assessment , 2011 .

[180]  Roman V. Yampolskiy,et al.  Artificial Intelligence Safety Engineering: Why Machine Ethics Is a Wrong Approach , 2011, PT-AI.

[181]  N Wiener,et al.  Some moral and technical consequences of automation , 1960, Science.

[182]  Patrick Lin,et al.  Moral Machines: Contradiction in Terms or Abdication of Human Responsibility? , 2012 .

[183]  Peter Dayan,et al.  Models of value and choice. , 2012 .

[184]  Carl Shulman,et al.  Which Consequentialism ? Machine Ethics and Moral Divergence , 2013 .

[185]  J. Haidt.  The happiness hypothesis: Finding modern truth in ancient wisdom. , 2006 .

[186]  Rodney Brooks,et al.  I, Rodney Brooks, am a robot , 2008, IEEE Spectrum.

[187]  Joy Bill,et al.  Why the future doesn’t need us , 2003 .

[188]  Laura Crompton.  Unfit for the Future , 2014 .

[189]  Ben Goertzel,et al.  Why an Intelligence Explosion is Probable , 2012 .

[190]  William Daley,et al.  Mitigating potential hazards to humans from the development of intelligent machines , 2011 .

[191]  Diana F. Spears,et al.  Assuring the Behavior of Adaptive Agents , 2006 .

[192]  Sarah K. Murnen,et al.  The effect of experimental presentation of thin media images on body satisfaction: a meta-analytic review. , 2002, The International journal of eating disorders.

[193]  J. Tenenbaum,et al.  Theory-based Bayesian models of inductive learning and reasoning , 2006, Trends in Cognitive Sciences.

[194]  Wendell Wallach,et al.  Machine morality: bottom-up and top-down approaches for modelling human moral faculties , 2008, AI & SOCIETY.

[195]  K. Warwick.  Implications and consequences of robots with biological brains , 2010, Ethics and Information Technology.

[196]  Anders Sandberg,et al.  Implications of a Software-Limited Singularity , 2013 .

[197]  R. Salekin.  Treatment of child and adolescent psychopathy: Focusing on change. , 2010 .

[198]  Anders Sandberg,et al.  An Overview of Models of Technological Singularity , 2013 .

[199]  Edward W. Felten,et al.  Timing attacks on Web privacy , 2000, CCS.

[200]  Alice C. Parker,et al.  Challenges for Brain Emulation: Why is Building a Brain so Difficult? , 2012 .

[201]  L. Versenyi,et al.  Can Robots be Moral? , 1974, Ethics.

[202]  H. Valpola,et al.  COALESCING MINDS: BRAIN UPLOADING-RELATED GROUP MIND SCENARIOS , 2012 .

[203]  Jason Nolan,et al.  Sousveillance: Inventing and Using Wearable Computing Devices for Data Collection in Surveillance Environments. , 2002 .

[204]  David Lewis,et al.  Dispositional Theories of Value , 1989 .

[205]  Philippe Golle,et al.  On the Anonymity of Home/Work Location Pairs , 2009, Pervasive.

[206]  Roman V Yampolskiy,et al.  Responses to catastrophic AGI risk: a survey , 2014 .

[207]  Chuen-Tsai Sun,et al.  Toward the Human–Robot Co-Existence Society: On Safety Intelligence for Next Generation Robots , 2009, Int. J. Soc. Robotics.

[208]  Joel Waldfogel,et al.  Introduction , 2010, Inf. Econ. Policy.

[209]  Gabriel Hallevy.  The Criminal Liability of Artificial Intelligence Entities , 2010 .

[210]  David Benatar,et al.  Better never to have been : the harm of coming into existence , 2006 .

[211]  Luke Muehlhauser,et al.  The Singularity and Machine Ethics , 2012 .

[212]  Robert Axelrod,et al.  The Evolution of Strategies in the Iterated Prisoner's Dilemma , 2001 .

[213]  S. Lehman-Wilzig.  Frankenstein unbound: Towards a legal definition of artificial intelligence☆ , 1981 .

[214]  J. Gips.  Towards the ethical robot , 1995 .

[215]  F Chessa,et al.  Enough: staying human in an engineered age , 2004, Journal of Medical Ethics.

[216]  Selmer Bringsjord,et al.  On How to Build a Moral Machine , 2013 .

[217]  Mihail C. Roco,et al.  Nanoethics: The Ethical and Social Implications of Nanotechnology , 2007 .

[218]  Stefan Savage,et al.  Inside the Slammer Worm , 2003, IEEE Secur. Priv..

[219]  L. McCauley.  AI Armageddon and the Three Laws of Robotics , 2007, Ethics and Information Technology.

[220]  Daniel Dewey,et al.  Learning What to Value , 2011, AGI.