Beyond Kant and Bentham: How Ethical Theories are being used in Artificial Moral Agents

From robots to autonomous vehicles, the design of artificial moral agents (AMAs) is an area of growing interest to HCI, and rising concern for the implications of deploying moral machines raises questions about which ethical frameworks are being used in AMAs. We performed a literature review to identify and synthesize key themes in how ethical theories have been applied to AMAs. We reviewed 53 papers and performed a thematic analysis to describe and synthesize the current conversation across HCI. We found many describe the value of ethical theories and implement them as technical contributions, but very few conduct empirical studies in real settings. Furthermore, we found AMA development is dominated by two ethical theories: deontology and consequentialism. We argue that the focus on deontology and consequentialism risks creating AMAs based on a narrow set of Western ethical values and concepts at the expense of other forms of moral reasoning.

[1]  Janet Davis,et al.  Make and Take an Ethics Module: Ethics Across the CS Curriculum , 2019, SIGCSE.

[2]  Konstantine Arkoudas,et al.  Toward Ethical Robots via Mechanized Deontic Logic ∗ , 2005 .

[3]  Jean-Gabriel Ganascia,et al.  Non-monotonic Resolution of Conflicts for Ethical Reasoning , 2015, A Construction Manual for Robots' Ethical Systems.

[4]  Matthias Scheutz,et al.  Beyond Moral Dilemmas: Exploring the Ethical Landscape in HRI , 2017, 2017 12th ACM/IEEE International Conference on Human-Robot Interaction (HRI.

[5]  Jean-Gabriel Ganascia,et al.  Modelling Moral Reasoning and Ethical Responsibility with Logic Programming , 2015, LPAR.

[6]  Alexander G. Mirnig,et al.  Trolled by the Trolley Problem: On What Matters for Ethical Decision Making in Automated Vehicles , 2019, CHI.

[7]  Selmer Bringsjord,et al.  On Automating the Doctrine of Double Effect , 2017, IJCAI.

[8]  Qin Zhu.  Computer Ethics - Philosophical Enquiry (CEPE) Proceedings , 2022 .

[9]  Felix Lindner,et al.  The Hybrid Ethical Reasoning Agent IMMANUEL , 2017, HRI.

[10]  Marcello Guarini.  Conative Dimensions of Machine Ethics: A Defense of Duty , 2012, IEEE Transactions on Affective Computing.

[11]  C. Allen,et al.  Moral Machines: Teaching Robots Right from Wrong , 2008 .

[12]  Christopher Grau,et al.  There Is No "I" in "Robot": Robots and Utilitarianism , 2006, IEEE Intelligent Systems.

[13]  J. Bentham.  An Introduction to the Principles of Morals and Legislation , 1945, Princeton Readings in Political Thought.

[14]  John N. Hooker,et al.  Toward Non-Intuition-Based Machine and Artificial Intelligence Ethics: A Deontological Approach Based on Modal Logic , 2018, AIES.

[15]  Selmer Bringsjord,et al.  Toward a General Logicist Methodology for Engineering Ethically Correct Robots , 2006, IEEE Intelligent Systems.

[16]  Kenneth D. Forbus,et al.  An Integrated Reasoning Approach to Moral Decision-Making , 2008, AAAI.

[17]  Martin Mose Bentzen,et al.  A Formalization of Kant's Second Formulation of the Categorical Imperative , 2018, DEON.

[18]  H. Acton,et al.  Kant's Moral Philosophy , 1970 .

[19]  Vincent Bonnemains,et al.  Embedded ethics: some technical and ethical challenges , 2018, Ethics and Information Technology.

[20]  Barry A. T. Brown,et al.  Against Ethical AI , 2019, HTTF.

[21]  Tsuyoshi Murata,et al.  {m , 1934, ACML.

[22]  Colin Allen,et al.  E TH E L : Toward a Principled Ethical Eldercare Robot , 2022 .

[23]  Ryan Tonkens,et al.  A Challenge for Machine Ethics , 2009, Minds and Machines.

[24]  Angela D. R. Smith,et al.  Critical Race Theory for HCI , 2020, CHI.

[25]  Iyad Rahwan,et al.  A Computational Model of Commonsense Moral Decision Making , 2018, AIES.

[26]  Wilhelm E. J. Klein.  Robots make ethics honest , 2016 .

[27]  Luis-Felipe Rodríguez,et al.  Autonomous Agents and Ethical Decision-Making , 2016, Cognitive Computation.

[28]  M. Clarke,et al.  The Doctrine of Double Effect , 2020, Forum Philosophicum.

[29]  Stephen C. Wingreen,et al.  Virtue Ethics as a Solution to the Privacy Paradox and Trust in Emerging Technologies , 2020, ICISS.

[30]  Ava Thomas Wright.  A Deontic Logic for Programming Rightful Machines , 2020, AIES.

[31]  Jan van den Berg,et al.  Combining Moral Theory, Modal Logic and Mas to Create Well-Behaving Artificial Agents , 2009, Int. J. Soc. Robotics.

[32]  Matthew Chalmers,et al.  Improving consent in large scale mobile HCI through personalised representations of data , 2014, NordiCHI.

[33]  Cynthia L. Bennett,et al.  What is the point of fairness? , 2019, ACM SIGACCESS Access. Comput..

[34]  Felix Lindner,et al.  An Explorative Comparison of Blame Attributions to Companion Robots Across Various Moral Dilemmas , 2018, HAI.

[35]  S. Merz.  Race after technology. Abolitionist tools for the new Jim Code , 2020, Ethnic and Racial Studies.

[36]  Lee Gillam,et al.  Machine Ethics for Metaverse Gambling: No Stake in a $24m Market? , 2009, 2009 Conference in Games and Virtual Worlds for Serious Applications.

[37]  Benjamin Kuipers,et al.  How can we trust a robot? , 2018, Commun. ACM.

[38]  Tom Williams,et al.  Towards A Role Ethics Approach to Command Rejection , 2019 .

[39]  Yngve Sundblad,et al.  Co-operative Design — perspectives on 20 years with 'the Scandinavian IT Design Model' , 2000 .

[40]  Shaowen Bardzell,et al.  Social Justice and Design: Power and oppression in collaborative systems , 2017, CSCW Companion.

[41]  Phoebe Sengers,et al.  Mapping the landscape of sustainable HCI , 2010, CHI.

[42]  Liz Lane.  Ethics of Activist Design: Designing a Technical Communication Pedagogy Grounded in Civic Engagement Activism , 2016, SIGDOC.

[43]  Ilse Verdiesen,et al.  The Design of Human Oversight in Autonomous Weapon Systems , 2018, AIES.

[44]  Kristian Kersting,et al.  Semantics Derived Automatically from Language Corpora Contain Human-like Moral Choices , 2019, AIES.

[45]  Shaowen Bardzell,et al.  The Proper Care and Feeding of Hackerspaces: Care Ethics and Cultures of Making , 2015, CHI.

[46]  Olivier Boissier,et al.  Ethical Judgment of Agents' Behaviors in Multi-Agent Systems , 2016, AAMAS.

[47]  Thomas M. Powers.  Prospects for a Kantian Machine , 2006, IEEE Intelligent Systems.

[48]  Colin Allen,et al.  Prolegomena to any future artificial moral agent , 2000, J. Exp. Theor. Artif. Intell..

[49]  Iyad Rahwan,et al.  Autonomous Vehicles Need Experimental Ethics: Are We Ready for Utilitarian Cars? , 2015, ArXiv.

[50]  Ava Thomas Wright,et al.  Rightful Machines and Dilemmas , 2019, AIES.

[51]  Gervase Vernon,et al.  Virtue ethics. , 2003, The British journal of general practice : the journal of the Royal College of General Practitioners.

[52]  Madeleine Akrich,et al.  The De-scription of Technical Objects , 1992 .

[53]  C. Allen,et al.  Artificial Morality: Top-down, Bottom-up, and Hybrid Approaches , 2005, Ethics and Information Technology.

[54]  Suvradip Maitra,et al.  Artificial Intelligence and Indigenous Perspectives: Protecting and Empowering Intelligent Human Beings , 2020, AIES.

[55]  Tom Williams,et al.  Blame-Laden Moral Rebukes and the Morally Competent Robot: A Confucian Ethical Perspective , 2020, Sci. Eng. Ethics.

[56]  Bernhard Nebel,et al.  The HERA approach to morally competent robots , 2017, 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS).

[57]  Jes A. Koepfler,et al.  Making space for values: communication & values levers in a virtual team , 2013, C&T '13.

[58]  Francesca Rossi,et al.  Embedding Ethical Principles in Collective Decision Support Systems , 2016, AAAI.

[59]  Gerhard Weikum,et al.  Fides: Towards a Platform for Responsible Data Science , 2017, SSDBM.

[60]  Lily Hu.  Justice Beyond Utility in Artificial Intelligence , 2018, AIES.

[61]  Martin Mose Bentzen,et al.  The Principle of Double Effect Applied to Ethical Dilemmas of Social Robots , 2016, Robophilosophy/TRANSOR.

[62]  Vincent Berenz,et al.  A Value Driven Agent: Instantiation of a Case-Supported Principle-Based Behavior Paradigm , 2017, AAAI Workshops.

[63]  G. NaveenSundar,et al.  Toward the Engineering of Virtuous Machines , 2018, AIES.

[64]  Meredith Durbin,et al.  A Mulching Proposal: Analysing and Improving an Algorithmic System for Turning the Elderly into High-Nutrient Slurry , 2019, CHI Extended Abstracts.

[65]  André Vellino,et al.  Ethical Decision Making in Robots: Autonomy, Trust and Responsibility - Autonomy Trust and Responsibility , 2016, ICSR.

[66]  Samuel B. Williams,et al.  ASSOCIATION FOR COMPUTING MACHINERY , 2000 .

[67]  Keith W. Miller,et al.  Un-making artificial moral agents , 2008, Ethics and Information Technology.

[68]  Paul Dourish,et al.  Postcolonial computing: a lens on design and development , 2010, CHI.

[69]  Katie Shilton,et al.  Values and Ethics in Human-Computer Interaction , 2018, Found. Trends Hum. Comput. Interact..

[70]  Petros Terzis.  Onward for the freedom of others: marching beyond the AI ethics , 2020, FAT*.

[71]  Ghislaine M. Lawrence.  The social construction of technological systems: new directions in the sociology and history of technology , 1989, Medical History.

[72]  Luis-Felipe Rodríguez,et al.  Artificial Moral Agents: A Survey of the Current Status , 2019, Science and Engineering Ethics.

[73]  J. Gips.  Towards the ethical robot , 1995 .

[74]  Pelle Ehn,et al.  Scandinavian Design: On Participation and Skill , 1992, Usability - Turning Technologies into Tools.

[75]  Shaowen Bardzell,et al.  Feminist HCI: taking stock and outlining an agenda for design , 2010, CHI.

[76]  Keith J. Burnham,et al.  Model-to-Decision Approach for Autonomous Vehicle Convoy Collision Ethics , 2018, 2018 UKACC 12th International Conference on Control (CONTROL).

[77]  Mario Romero,et al.  Situational Ethics: Re-thinking Approaches to Formal Ethics Requirements for Human-Computer Interaction , 2015, CHI.

[78]  Jean-Gabriel Ganascia,et al.  A Declarative Modular Framework for Representing and Applying Ethical Principles , 2017, AAMAS.

[79]  Michael Anderson,et al.  Toward ethical intelligent autonomous healthcare agents : a case-supported principle-based behavior paradigm , 2014 .

[80]  Michael Anderson,et al.  An Approach to Computing Ethics , 2006, IEEE Intelligent Systems.

[81]  Chris Armen,et al.  Towards Machine Ethics: Implementing Two Action-Based Ethical Theories , 2005 .

[82]  Cindy L. Bethel,et al.  Proceedings of the Workshop on Social Robots in Therapy: Focusing on Autonomy and Ethical Challenges , 2018, ArXiv.

[83]  Helen Nissenbaum,et al.  Software agents and user autonomy , 1997, AGENTS '97.

[84]  J. Tronto,et al.  Toward a Feminist Theory of Caring , 1990 .

[85]  Michael Anderson,et al.  GenEth: a general ethical dilemma analyzer , 2014, AAAI.

[86]  Feiyang Sun,et al.  Beyond Open vs. Closed: Balancing Individual Privacy and Public Accountability in Data Sharing , 2019, FAT.

[87]  Qin Zhu,et al.  The Confucian Matador: Three Defenses Against the Mechanical Bull , 2020, HRI.

[88]  Selmer Bringsjord,et al.  Beyond the Doctrine of Double Effect: A Formal Model of True Self-sacrifice , 2019, Robotics and Well-Being.

[89]  Michael Anderson,et al.  ETHEL: Toward a Principled Ethical Eldercare System , 2008, AAAI Fall Symposium: AI in Eldercare: New Solutions to Old Problems.

[90]  L. Yilmaz.  Verification and validation of ethical decision-making in autonomous systems , 2017 .

[91]  Michael J. Heron,et al.  Fuzzy ethics: or how I learned to stop worrying and love the bot , 2015, CSOC.

[92]  Luís Moniz Pereira,et al.  Modelling morality with prospective logic , 2007, Int. J. Reason. based Intell. Syst..

[93]  Michael Anderson,et al.  A Prima Facie Duty Approach to Machine Ethics and Its Application to Elder Care , 2011, Human-Robot Interaction in Elder Care.

[94]  Marija Slavkovik,et al.  On the Distinction between Implicit and Explicit Ethical Agency , 2018, AIES.

[95]  e-traces,et al.  Self-Driving Uber Car Kills Pedestrian in Arizona, Where Robots Roam - e-traces , 2018 .

[96]  Jess Whittlestone,et al.  The Role and Limits of Principles in AI Ethics: Towards a Focus on Tensions , 2019, AIES.