Algorithmic decision-making systems are increasingly being deployed to make, or to support humans to make, decisions that impact people’s lives in significant ways. Yet, decision subjects, those affected by algorithmic decisions, can be limited in their ability to contest these decisions. For example, the Education Value-Added Assessment System (EVAAS), a statistical method used to predict academic growth, was used by the Houston Independent School District to evaluate teachers’ performance and, in a number of cases, to terminate teachers’ contracts. Twelve teachers and the Houston Federation of Teachers successfully argued in court that the teachers’ constitutional right to due process was violated because they were unable to contest, or ‘meaningfully challenge’, the termination of their contracts because there was a ‘lack of sufficient information’ — the private company that designed EVAAS would not release the source codes or methodology used as they were proprietary trade secrets [1]. Even for decision subjects who are able to understand why a decision has been made and are provided with means to contest that decision, contestation systems can be seen as severely lacking [12]. Sarah Myers West studied content moderation across a number of social media platforms, most of which offered a way for users to contest a decision to remove their content from the platform [12]. Myers West reported user dissatisfaction with the contestation systems for a number of reasons, including a lack of clear instruction about how to lodge an appeal, no reply being received, no resolution being reached after a challenge has been lodged, and a lack of access to human intervention. These examples demonstrate that being able to challenge algorithmic decisions is important to decision subjects, yet numerous factors can limit a person’s ability to contest such decisions. We propose that administrative law systems, which were created to ensure that governments are kept accountable for their actions and decision making in relation to individuals [5, 6], can provide guidance on how to design contestation systems for algorithmic decision-making. There are similarities between government decision-making and algorithmic decision-making that suggest there is value in considering how the administrative law system enables contestation. For example, in both cases decision making can be said to occur ‘behind closed doors’, which limits
[1]
I. Rahwan,et al.
Why we need to audit algorithms
,
2018
.
[2]
Sarah Myers West,et al.
Censored, suspended, shadowbanned: User interpretations of content moderation on social media platforms
,
2018,
New Media Soc..
[3]
Carlos Guestrin,et al.
"Why Should I Trust You?": Explaining the Predictions of Any Classifier
,
2016,
ArXiv.
[4]
Jun Zhao,et al.
'It's Reducing a Human Being to a Percentage': Perceptions of Justice in Algorithmic Decisions
,
2018,
CHI.
[5]
Amina Adadi,et al.
Peeking Inside the Black-Box: A Survey on Explainable Artificial Intelligence (XAI)
,
2018,
IEEE Access.
[6]
Chris Russell,et al.
Counterfactual Explanations Without Opening the Black Box: Automated Decisions and the GDPR
,
2017,
ArXiv.
[7]
Mohan S. Kankanhalli,et al.
Trends and Trajectories for Explainable, Accountable and Intelligible Systems: An HCI Research Agenda
,
2018,
CHI.
[8]
Qian Yang,et al.
Designing Theory-Driven User-Centric Explainable AI
,
2019,
CHI.
[9]
Tim Miller,et al.
Explainable AI: Beware of Inmates Running the Asylum Or: How I Learnt to Stop Worrying and Love the Social and Behavioural Sciences
,
2017,
ArXiv.
[10]
M. Borzycki.
Interventions for prisoners returning to the community: a report prepared by the Australian Institute of Criminology for the Community Safety and Justice Branch of the Australian Government Attorney-General’s Department
,
2005
.
[11]
Shrikanth S. Narayanan,et al.
Designing Contestability: Interaction Design, Machine Learning, and Mental Health
,
2017,
Conference on Designing Interactive Systems.
[12]
Marco Almada.
Human Intervention in Automated Decision-Making: Toward the Construction of Contestable Systems
,
2018
.