Scruples: A Corpus of Community Ethical Judgments on 32, 000 Real-Life Anecdotes

As AI systems become an increasing part of people's everyday lives, it becomes ever more important that they understand people's ethical norms. Motivated by descriptive ethics, a field of study that focuses on people's descriptive judgments rather than theoretical prescriptions on morality, we investigate a novel, data-driven approach to machine ethics. We introduce Scruples, the first large-scale dataset with 625,000 ethical judgments over 32,000 real-life anecdotes. Each anecdote recounts a complex ethical situation, often posing moral dilemmas, paired with a distribution of judgments contributed by the community members. Our dataset presents a major challenge to state-of-the-art neural language models, leaving significant room for improvement. However, when presented with simplified moral situations, the results are considerably more promising, suggesting that neural models can effectively learn simpler ethical building blocks. A key take-away of our empirical analysis is that norms are not always clean-cut; many situations are naturally divisive. We present a new method to estimate the best possible performance on such tasks with inherently diverse label distributions, and explore likelihood functions that separate intrinsic from model uncertainty.

[1]  Yarin Gal,et al.  Uncertainty in Deep Learning , 2016 .

[2]  Mark O. Riedl,et al.  Learning Norms from Stories: A Prior for Value Aligned Agents , 2019, AIES.

[3]  Tianqi Chen,et al.  XGBoost: A Scalable Tree Boosting System , 2016, KDD.

[4]  C. Allen,et al.  Stanford Encyclopedia of Philosophy , 2011 .

[5]  Leo Breiman,et al.  Random Forests , 2001, Machine Learning.

[6]  Benjamin Van Durme,et al.  Reporting bias and knowledge acquisition , 2013, AKBC '13.

[7]  Omer Levy,et al.  Annotation Artifacts in Natural Language Inference Data , 2018, NAACL.

[8]  Hung-Yu Kao,et al.  Probing Neural Network Comprehension of Natural Language Arguments , 2019, ACL.

[9]  Ole Tange,et al.  GNU Parallel: The Command-Line Power Tool , 2011, login Usenix Mag..

[10]  David R. Karger,et al.  Tackling the Poor Assumptions of Naive Bayes Text Classifiers , 2003, ICML.

[11]  Ellie Pavlick,et al.  Inherent Disagreements in Human Textual Inferences , 2019, Transactions of the Association for Computational Linguistics.

[12]  Gaël Varoquaux,et al.  Scikit-learn: Machine Learning in Python , 2011, J. Mach. Learn. Res..

[13]  Udo Kruschwitz,et al.  Comparing Bayesian Models of Annotation , 2018, TACL.

[14]  Kilian Q. Weinberger,et al.  On Calibration of Modern Neural Networks , 2017, ICML.

[15]  Keith W. Miller,et al.  Why we should have seen that coming: comments on Microsoft's tay "experiment," and wider implications , 2017, CSOC.

[16]  Francesca Rossi,et al.  Incorporating Behavioral Constraints in Online AI Systems , 2018, AAAI.

[17]  Selmer Bringsjord,et al.  Toward a General Logicist Methodology for Engineering Ethically Correct Robots , 2006, IEEE Intelligent Systems.

[18]  John Schulman,et al.  Concrete Problems in AI Safety , 2016, ArXiv.

[19]  Susan Leigh Anderson,et al.  Machine ethics : papers from the AAAI Fall Symposium , 2005 .

[20]  Shou-De Lin,et al.  A Low-Cost Ethics Shaping Approach for Designing Reinforcement Learning Agents , 2017, AAAI.

[21]  Kevin P. Murphy,et al.  Machine learning - a probabilistic perspective , 2012, Adaptive computation and machine learning series.

[22]  Aida Mostafazadeh Davani,et al.  Moral Foundations Twitter Corpus: A Collection of 35k Tweets Annotated for Moral Sentiment , 2020 .

[23]  Michael Wiegand,et al.  A Survey on Hate Speech Detection using Natural Language Processing , 2017, SocialNLP@EACL.

[24]  V. Vinge.  The coming technological singularity , 1993 .

[25]  Kyriaki Kalimeri,et al.  MoralStrength: Exploiting a Moral Lexicon and Embedding Similarity for Moral Foundations Prediction , 2019, Knowl. Based Syst..

[26]  Brian A. Nosek,et al.  Liberals and conservatives rely on different sets of moral foundations. , 2009, Journal of personality and social psychology.

[27]  Kristian Kersting,et al.  BERT has a Moral Compass: Improvements of ethical and moral values of machines , 2019, ArXiv.

[28]  Francesca Rossi,et al.  Building Ethically Bounded AI , 2018, AAAI.

[29]  Alex S. Taylor,et al.  Let's Talk About Race: Identity, Chatbots, and AI , 2018, CHI.

[30]  Chunyan Miao,et al.  Building Ethics into Artificial Intelligence , 2018, IJCAI.

[31]  R. Lewis,et al.  A question of responsibility. , 1990, Australian family physician.

[32]  John Skorupski,et al.  The Definition of Morality , 1993, Royal Institute of Philosophy Supplement.

[33]  Eliezer Yudkowsky.  Artificial Intelligence as a Positive and Negative Factor in Global Risk , 2006 .

[34]  Omer Levy,et al.  RoBERTa: A Robustly Optimized BERT Pretraining Approach , 2019, ArXiv.

[35]  Michael Anderson,et al.  Machine Ethics: Creating an Ethical Intelligent Agent , 2007, AI Mag..

[36]  Ming-Wei Chang,et al.  BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding , 2019, NAACL.

[37]  Michael I. Jordan,et al.  Latent Dirichlet Allocation , 2001, J. Mach. Learn. Res..

[38]  Colin Raffel,et al.  Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer , 2019, J. Mach. Learn. Res..

[39]  Luca Antiga,et al.  Automatic differentiation in PyTorch , 2017 .

[40]  C. Allen,et al.  Moral Machines: Teaching Robots Right from Wrong , 2008 .

[41]  Kristian Kersting,et al.  Semantics Derived Automatically from Language Corpora Contain Human-like Moral Choices , 2019, AIES.

[42]  Masatoshi Tsuchiya,et al.  Performance Impact Caused by Hidden Bias of Training Data for Recognizing Textual Entailment , 2018, LREC.

[43]  Fiona Steele,et al.  The Analysis and Interpretation of Multivariate Data for Social Scientists , 2002 .

[44]  Kevin Gimpel,et al.  ALBERT: A Lite BERT for Self-supervised Learning of Language Representations , 2019, ICLR.

[45]  Adam Tauman Kalai,et al.  Man is to Computer Programmer as Woman is to Homemaker? Debiasing Word Embeddings , 2016, NIPS.

[46]  Oren Etzioni,et al.  The First Law of Robotics (A Call to Arms) , 1994, AAAI.

[47]  Dirk Hovy,et al.  Learning Whom to Trust with MACE , 2013, NAACL.

[48]  Michael L. Littman,et al.  Reinforcement Learning as a Framework for Ethical Decision Making , 2016, AAAI Workshop: AI, Ethics, and Society.

[49]  Thomas Wolf,et al.  HuggingFace's Transformers: State-of-the-art Natural Language Processing , 2019, ArXiv.

[50]  J. Haidt,et al.  Intuitive ethics: how innately prepared intuitions generate culturally variable virtues , 2004, Daedalus.

[51]  Jeremy Blackburn,et al.  The Pushshift Reddit Dataset , 2020, ICWSM.

[52]  I. J. Good,et al.  Speculations Concerning the First Ultraintelligent Machine , 1965, Adv. Comput..

[53]  Peter H. Ditto,et al.  Moral Foundations Theory: The Pragmatic Validity of Moral Pluralism , 2012 .

[54]  Rachel Rudinger,et al.  Hypothesis Only Baselines in Natural Language Inference , 2018, *SEMEVAL.

[55]  D. Murphey,et al.  The Righteous Mind: Why Good People Are Divided by Politics and Religion , 2013 .

[56]  K. Araki,et al.  Aristotelian Approach and Shallow Search Settings for Fast Ethical Judgment , 2013 .

[57]  I. Watson,et al.  Moral preferences , 2016 .

[58]  Stuart J. Russell,et al.  Research Priorities for Robust and Beneficial Artificial Intelligence , 2015, AI Mag..

[59]  Eric R. Ziegel,et al.  Generalized Linear Models , 2002, Technometrics.

[60]  S. Holm.  A Simple Sequentially Rejective Multiple Test Procedure , 1979 .

[61]  David Ginsbourger,et al.  Fast Computation of the Multi-Points Expected Improvement with Applications in Batch Selection , 2013, LION.

[62]  Zoubin Ghahramani,et al.  Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning , 2015, ICML.