XAI tools in the public sector: a case study on predicting combined sewer overflows

Artificial intelligence and deep learning are becoming increasingly prevalent in contemporary software solutions. Explainable artificial intelligence (XAI) tools attempt to address the black box nature of the deep learning models and make them more understandable to humans. In this work, we apply three state-of-the-art XAI tools in a real-world case study. Our study focuses on predicting combined sewer overflow events for a municipal wastewater treatment organization. Through a data driven inquiry, we collect both qualitative information via stakeholder interviews and quantitative measures. These help us assess the predictive accuracy of the XAI tools, as well as the simplicity, soundness, and insightfulness of the produced explanations. Our results not only show the varying degrees that the XAI tools meet the requirements, but also highlight that domain experts can draw new insights from complex explanations that may differ from their previous expectations.

[1]  Nan Niu,et al.  Using soft systems methodology to improve requirements practices: an exploratory case study , 2011, IET Softw..

[2]  Jian Sun,et al.  Deep Residual Learning for Image Recognition , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[3]  Amina Adadi,et al.  Peeking Inside the Black-Box: A Survey on Explainable Artificial Intelligence (XAI) , 2018, IEEE Access.

[4]  Yoshua Bengio,et al.  Understanding the difficulty of training deep feedforward neural networks , 2010, AISTATS.

[5]  Nan Niu,et al.  So, You Think You Know Others' Goals? A Repertory Grid Study , 2007, IEEE Software.

[6]  Kurt Schneider,et al.  Explainability as a non-functional requirement: challenges and recommendations , 2020, Requirements Engineering.

[7]  Anna Jobin,et al.  The global landscape of AI ethics guidelines , 2019, Nature Machine Intelligence.

[8]  Christine T. Wolf,et al.  Explainability in Context: Lessons from an Intelligent System in the IT Services Domain , 2019, IUI Workshops.

[9]  Nan Niu,et al.  Deep Learning for Smart Sewer Systems: Assessing Nonfunctional Requirements , 2020, 2020 IEEE/ACM 42nd International Conference on Software Engineering: Software Engineering in Society (ICSE-SEIS).

[10]  John Grundy,et al.  Explainable AI for Software Engineering , 2020, 2021 36th IEEE/ACM International Conference on Automated Software Engineering (ASE).

[11]  Tim Miller,et al.  Explainable AI: Beware of Inmates Running the Asylum Or: How I Learnt to Stop Worrying and Love the Social and Behavioural Sciences , 2017, ArXiv.

[12]  Sjaak Brinkkemper,et al.  Requirements Engineering and Continuous Deployment , 2018, IEEE Software.

[13]  Seth Flaxman,et al.  European Union Regulations on Algorithmic Decision-Making and a "Right to Explanation" , 2016, AI Mag..

[14]  Hui Liu,et al.  Enhancing Automated Requirements Traceability by Resolving Polysemy , 2018, 2018 IEEE 26th International Requirements Engineering Conference (RE).

[15]  Nan Niu,et al.  Visual requirements analytics: a framework and case study , 2013, Requirements Engineering.

[16]  Franco Turini,et al.  A Survey of Methods for Explaining Black Box Models , 2018, ACM Comput. Surv..

[17]  Avanti Shrikumar,et al.  Learning Important Features Through Propagating Activation Differences , 2017, ICML.

[18]  D. Hilton.  Conversational processes and causal explanation. , 1990 .

[19]  Alexander Binder,et al.  On Pixel-Wise Explanations for Non-Linear Classifier Decisions by Layer-Wise Relevance Propagation , 2015, PloS one.

[20]  Huamin Qu,et al.  RuleMatrix: Visualizing and Understanding Classifiers with Rules , 2018, IEEE Transactions on Visualization and Computer Graphics.

[21]  John Mylopoulos,et al.  Non-Functional Requirements in Software Engineering , 2000, International Series in Software Engineering.

[22]  Been Kim,et al.  Towards A Rigorous Science of Interpretable Machine Learning , 2017, 1702.08608.

[23]  Ignacio Requena,et al.  Are artificial neural networks black boxes? , 1997, IEEE Trans. Neural Networks.

[24]  Mark Harman,et al.  Machine Learning Testing: Survey, Landscapes and Horizons , 2019, IEEE Transactions on Software Engineering.

[25]  Andrew Zisserman,et al.  Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Maps , 2013, ICLR.

[26]  Nan Niu,et al.  Keeping requirements on track via visual analytics , 2013, 2013 21st IEEE International Requirements Engineering Conference (RE).

[27]  Linda Newman,et al.  Advancing viewpoint merging in requirements engineering: a theoretical replication and explanatory study , 2017, Requirements Engineering.

[28]  Vander Alves,et al.  An Exploratory Case Study on Exploiting Aspect Orientation in Mobile Game Porting , 2013, IRI.

[29]  Scott Lundberg,et al.  A Unified Approach to Interpreting Model Predictions , 2017, NIPS.

[30]  Carlos Guestrin,et al.  "Why Should I Trust You?": Explaining the Predictions of Any Classifier , 2016, ArXiv.

[31]  P. Thagard,et al.  Explanatory coherence , 1993 .

[32]  G. Talamini,et al.  Combined sewer overflow in Shenzhen, China: the case study of Dasha River , 2016 .

[33]  G. Crooks.  On Measures of Entropy and Information , 2015 .

[34]  Linda Newman,et al.  Advancing Repeated Research in Requirements Engineering: A Theoretical Replication of Viewpoint Merging , 2016, 2016 IEEE 24th International Requirements Engineering Conference (RE).

[35]  Dimitri Bohlender,et al.  Explainability as a Non-Functional Requirement , 2019, 2019 IEEE 27th International Requirements Engineering Conference (RE).

[36]  Alun D. Preece,et al.  Interpretability of deep learning models: A survey of results , 2017, 2017 IEEE SmartWorld, Ubiquitous Intelligence & Computing, Advanced & Trusted Computed, Scalable Computing & Communications, Cloud & Big Data Computing, Internet of People and Smart City Innovation (SmartWorld/SCALCOM/UIC/ATC/CBDCom/IOP/SCI).

[37]  Fabiano Dalpiaz,et al.  Requirements Engineering in the Days of Artificial Intelligence , 2020, IEEE Software.

[38]  Zhendong Niu,et al.  Analysis of Architecturally Significant Requirements for Enterprise Systems , 2014, IEEE Systems Journal.

[39]  Gaël Varoquaux,et al.  Scikit-learn: Machine Learning in Python , 2011, J. Mach. Learn. Res..

[40]  Lalana Kagal,et al.  Explaining Explanations: An Overview of Interpretability of Machine Learning , 2018, 2018 IEEE 5th International Conference on Data Science and Advanced Analytics (DSAA).

[41]  Jimmy Ba,et al.  Adam: A Method for Stochastic Optimization , 2014, ICLR.

[42]  Harald C. Gall,et al.  Software Engineering for Machine Learning: A Case Study , 2019, 2019 IEEE/ACM 41st International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP).

[43]  Nan Niu,et al.  Faulty Requirements Made Valuable: On the Role of Data Quality in Deep Learning , 2020, 2020 IEEE Seventh International Workshop on Artificial Intelligence for Requirements Engineering (AIRE).

[44]  Impacts and Control of CSOs and SSOs , 2022 .

[45]  T. Lombrozo,et al.  Simplicity and probability in causal explanation , 2007, Cognitive Psychology.

[46]  Aditya K. Ghose,et al.  Explainable Software Analytics , 2018, 2018 IEEE/ACM 40th International Conference on Software Engineering: New Ideas and Emerging Technologies Results (ICSE-NIER).

[47]  Christine Nadel,et al.  Case Study Research Design And Methods , 2016 .

[48]  Christine T. Wolf,et al.  Evaluating the Promise of Human-Algorithm Collaborations in Everyday Work Practices , 2019, Proc. ACM Hum. Comput. Interact..

[49]  Egon Berghout,et al.  The Goal/Question/Metric method: a practical guide for quality improvement of software development , 1999 .