Reproducibility as a Mechanism for Teaching Fairness, Accountability, Confidentiality, and Transparency in Artificial Intelligence

In this work, we explain the setup for a technical, graduatelevel course on Fairness, Accountability, Confidentiality, and Transparency in Artificial Intelligence (FACT-AI) at the University of Amsterdam, which teaches FACT-AI concepts through the lens of reproducibility. The focal point of the course is a group project based on reproducing existing FACT-AI algorithms from top AI conferences and writing a corresponding report. In the first iteration of the course, we created an open source repository with the code implementations from the group projects. In the second iteration, we encouraged students to submit their group projects to the Machine Learning Reproducibility Challenge, resulting in 9 reports from our course being accepted for publication in the ReScience journal. We reflect on our experience teaching the course over two years, where one year coincided with a global pandemic, and propose guidelines for teaching FACTAI through reproducibility in graduate-level AI study programs. We hope this can be a useful resource for instructors who want to set up similar courses in the future.

[1]  Casey Fiesler,et al.  What Do We Teach When We Teach Tech Ethics?: A Syllabi Analysis , 2020, SIGCSE.

[2]  Ana Lucic,et al.  Order in the Court: Explainable AI Methods Prone to Disagreement , 2021, ArXiv.

[3]  Nancy L. GREEN,et al.  Argument Schemes for AI Ethics Education , 2020, CMNA@COMMA.

[4]  Inioluwa Deborah Raji,et al.  You Can't Sit With Us: Exclusionary Pedagogy in AI Ethics Education , 2021, FAccT.

[5]  Casey Fiesler,et al.  More Than "If Time Allows": The Role of Ethics in AI Education , 2020, AIES.

[6]  Robert Heckman,et al.  Integrating Ethics within Machine Learning Courses , 2019, ACM Trans. Comput. Educ..

[7]  Sabina Leonelli,et al.  Locating ethics in data science: responsibility and accountability in global and distributed knowledge production systems , 2016, Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences.

[8]  Nancy Green,et al.  An AI Ethics Course Highlighting Explicit Ethical Agents , 2021, AIES.

[9]  Fred Martin,et al.  Introducing Ethical Thinking About Autonomous Vehicles Into an AI Course , 2018, AAAI.

[10]  D. Sculley,et al.  Winner's Curse? On Pace, Progress, and Empirical Rigor , 2018, ICLR.

[11]  Zachary C. Lipton,et al.  Troubling Trends in Machine Learning Scholarship , 2018, ACM Queue.

[12]  Nathan Srebro,et al.  Equality of Opportunity in Supervised Learning , 2016, NIPS.

[13]  M. Hutson.  Artificial intelligence faces reproducibility crisis. , 2018, Science.

[14]  Ian Goodfellow,et al.  Deep Learning with Differential Privacy , 2016, CCS.

[15]  Carlos Guestrin,et al.  "Why Should I Trust You?": Explaining the Predictions of Any Classifier , 2016, ArXiv.

[16]  Haiyi Zhu,et al.  Value Cards: An Educational Toolkit for Teaching Social Impacts of Machine Learning through Deliberation , 2020, ArXiv.

[17]  Paul M. Leidig,et al.  Computing Competencies for Undergraduate Data Science Programs: an ACM Task Force Final Report , 2021, SIGCSE.

[18]  Julia Stoyanovich,et al.  Teaching Responsible Data Science: Charting New Pedagogical Territory , 2019, International Journal of Artificial Intelligence in Education.

[19]  Annibale Panichella,et al.  ReproducedPapers.org: Openly teaching and structuring machine learning reproducibility , 2021, RRPR.

[20]  Frank Hopfgartner,et al.  Integrating FATE/critical data studies into data science curricula: where are we going and how do we get there? , 2020, FAT*.

[21]  Srinjita Bhaduri,et al.  Ethics Education in Context: A Case Study of Novel Ethics Activities for the CS Classroom , 2018, SIGCSE.

[22]  Nicholas Mattei,et al.  How to teach computer ethics through science fiction , 2018, Commun. ACM.