Machine Reasoning Explainability

As a field of AI, Machine Reasoning (MR) uses largely symbolic means to formalize and emulate abstract reasoning. Studies in early MR have notably started inquiries into Explainable AI (XAI) -- arguably one of the biggest concerns today for the AI community. Work on explainable MR as well as on MR approaches to explainability in other areas of AI has continued ever since. It is especially potent in modern MR branches, such as argumentation, constraint and logic programming, planning. We hereby aim to provide a selective overview of MR explainability techniques and studies in hopes that insights from this long track of research will complement well the current XAI landscape. This document reports our work in-progress on MR explainability.

[1]  Tim Miller,et al.  Explanation in Artificial Intelligence: Insights from the Social Sciences , 2017, Artif. Intell..

[2]  Chris Russell,et al.  Explaining Explanations in AI , 2018, FAT.

[3]  Mohan Sridharan,et al.  Integrating Non-monotonic Logical Reasoning and Inductive Learning With Deep Learning for Explainable Visual Question Answering , 2019, Front. Robot. AI.

[4]  Murray Shanahan,et al.  Prediction is Deduction but Explanation is Abduction , 1989, IJCAI.

[5]  Xudong Luo,et al.  An explainable multi-attribute decision model based on argumentation , 2019, Expert Syst. Appl..

[6]  Subbarao Kambhampati,et al.  Explicable Planning as Minimizing Distance from Expected Behavior , 2019, AAMAS.

[7]  Tim Miller,et al.  Model-based contrastive explanations for explainable planning , 2019 .

[8]  Stefan Woltran,et al.  Explaining Non-Acceptability in Abstract Argumentation , 2020, ECAI.

[9]  Ned Hall.  Two Concepts of Causation , 2001 .

[10]  Francesca Toni,et al.  Deciding the Winner of a Debate Using Bipolar Argumentation , 2019, AAMAS.

[11]  Mykel J. Kochenderfer,et al.  Reinforcement Learning with Probabilistic Guarantees for Autonomous Driving , 2019, ArXiv.

[12]  Ajay Chander,et al.  Explanation Perspectives from the Cognitive Sciences - A Survey , 2020, IJCAI.

[13]  Krysia Broda,et al.  Logic-Based Learning of Answer Set Programs , 2019, Reasoning Web.

[14]  Peter A. Flach,et al.  One Explanation Does Not Fit All , 2020, KI - Künstliche Intelligenz.

[15]  Franco Turini,et al.  A Survey of Methods for Explaining Black Box Models , 2018, ACM Comput. Surv..

[16]  Andrea Omicini,et al.  An Abstract Framework for Agent-Based Explanations in AI , 2020, AAMAS.

[17]  Tamer Basar,et al.  Multi-Agent Reinforcement Learning: A Selective Overview of Theories and Algorithms , 2019, Handbook of Reinforcement Learning and Control.

[18]  Adnan Darwiche,et al.  A Symbolic Approach to Explaining Bayesian Network Classifiers , 2018, IJCAI.

[19]  Vasa Curcin,et al.  Computational argumentation-based clinical decision support: Demonstration , 2019, AAMAS 2019.

[20]  Miroslaw Truszczynski,et al.  Answer set programming at a glance , 2011, Commun. ACM.

[21]  A. Agogino,et al.  Challenges of Explaining Real-Time Planning , 2019 .

[22]  Gérard Ferrand,et al.  Explanations and Proof Trees , 2005, Comput. Artif. Intell..

[23]  Ringo Baumann,et al.  Expanding Argumentation Frameworks: Enforcing and Monotonicity Results , 2010, COMMA.

[24]  Eric M. S. P. Veith,et al.  Explainable Reinforcement Learning: A Survey , 2020, CD-MAKE.

[25]  Daniele Magazzeni,et al.  A New Approach to Plan-Space Explanation: Analyzing Plan-Property Dependencies in Oversubscription Planning , 2020, AAAI.

[26]  Francesca Toni,et al.  Automated information management via abductive logic agents , 2001, Telematics Informatics.

[27]  Thomas Lukasiewicz,et al.  Explanations for Inconsistency-Tolerant Query Answering under Existential Rules , 2020, AAAI.

[28]  Scott Lundberg,et al.  A Unified Approach to Interpreting Model Predictions , 2017, NIPS.

[29]  Or Biran,et al.  Explanation and Justification in Machine Learning : A Survey Or , 2017 .

[30]  Abdelraouf Hecham,et al.  An Empirical Evaluation of Argumentation in Explaining Inconsistency-Tolerant Query Answering , 2017, Description Logics.

[31]  Douglas Walton,et al.  A new dialectical theory of explanation , 2004 .

[32]  Claudette Cayrol,et al.  Change in Abstract Argumentation Frameworks: Adding an Argument , 2010, J. Artif. Intell. Res..

[33]  Francesca Toni,et al.  Explanatory predictions with artificial neural networks and argumentation , 2018 .

[34]  Phan Minh Dung,et al.  Dialectic proof procedures for assumption-based, admissible argumentation , 2006, Artif. Intell..

[35]  Sanjay Modgil,et al.  Proof Theories and Algorithms for Abstract Argumentation Frameworks , 2009, Argumentation in Artificial Intelligence.

[36]  Ringo Baumann.  What Does it Take to Enforce an Argument? Minimal Change in abstract Argumentation , 2012, ECAI.

[37]  Adnan Darwiche,et al.  Model-Based Diagnosis using Structured System Descriptions , 1998, J. Artif. Intell. Res..

[38]  Phan Minh Dung,et al.  On the Acceptability of Arguments and its Fundamental Role in Nonmonotonic Reasoning, Logic Programming and n-Person Games , 1995, Artif. Intell..

[39]  Christophe Labreuche,et al.  Explaining robust additive utility models by sequences of preference swaps , 2015, Theory and Decision.

[40]  Yves Bertot,et al.  Interactive Theorem Proving and Program Development: Coq'Art The Calculus of Inductive Constructions , 2010 .

[41]  Subbarao Kambhampati,et al.  The Emerging Landscape of Explainable Automated Planning & Decision Making , 2020, IJCAI.

[42]  Ufuk Topcu,et al.  Safe Reinforcement Learning via Shielding , 2017, AAAI.

[43]  Sandra Carberry,et al.  Techniques for Plan Recognition , 2001, User Modeling and User-Adapted Interaction.

[44]  Helmut Horacek.  How to Build Explanations of Automated Proofs: A Methodology and Requirements on Domain Representations , 2007, ExaCt.

[45]  Rachel K. E. Bellamy,et al.  Planning and visualization for a smart meeting room assistant , 2019, AI Commun..

[46]  Bernhard Nebel,et al.  Coming up With Good Excuses: What to do When no Plan Can be Found , 2010, Cognitive Robotics.

[47]  Francesca Toni,et al.  ABA-Based Answer Set Justification , 2013, Theory Pract. Log. Program..

[48]  Dimos V. Dimarogonas,et al.  On the Timed Temporal Logic Planning of Coupled Multi-Agent Systems , 2017, Autom..

[49]  autoepistemic Zogic.  Logic programming and negation : a survey , 2001 .

[50]  Bijan Parsia,et al.  Finding All Justifications of OWL DL Entailments , 2007, ISWC/ASWC.

[51]  Peter J. Stuckey,et al.  Explaining Propagators for String Edit Distance Constraints , 2020, AAAI.

[52]  Sebastian Junges,et al.  Shielded Decision-Making in MDPs , 2018, ArXiv.

[53]  Peter A. Flach,et al.  FACE: Feasible and Actionable Counterfactual Explanations , 2020, AIES.

[54]  Pascal Poupart,et al.  Minimal Sufficient Explanations for Factored Markov Decision Processes , 2009, ICAPS.

[55]  Farida Zehraoui,et al.  A new Transparent Ensemble Method based on Deep learning , 2019, KES.

[56]  Jörg Hoffmann,et al.  Explaining the Space of Plans through Plan-Property Dependencies , 2019 .

[57]  Richard Evans,et al.  Learning Explanatory Rules from Noisy Data , 2017, J. Artif. Intell. Res..

[58]  Guillermo Ricardo Simari,et al.  Argument-based mixed recommenders and their application to movie suggestion , 2014, Expert Syst. Appl..

[59]  Ernest Davis,et al.  Logical Formalizations of Commonsense Reasoning: A Survey , 2017, J. Artif. Intell. Res..

[60]  David Baxter,et al.  Interactive Natural Language Explanations of Cyc Inferences , 2005, ExaCt.

[61]  Elizabeth Sklar,et al.  Explainable Argumentation for Wellness Consultation , 2019, EXTRAAMAS@AAMAS.

[62]  Sanjay Modgil,et al.  Reasoning about preferences in argumentation frameworks , 2009, Artif. Intell..

[63]  David Heckerman,et al.  A Bayesian Approach to Learning Causal Networks , 1995, UAI.

[64]  Raymond Reiter,et al.  A Logic for Default Reasoning , 1987, Artif. Intell..

[65]  Carmen Lacave,et al.  A review of explanation methods for Bayesian networks , 2002, The Knowledge Engineering Review.

[66]  Peter A. Flach,et al.  Explainability fact sheets: a framework for systematic assessment of explainable approaches , 2019, FAT*.

[67]  Silvio Micali,et al.  The Knowledge Complexity of Interactive Proof Systems , 1989, SIAM J. Comput..

[68]  Matti Järvisalo,et al.  Smallest Explanations and Diagnoses of Rejection in Abstract Argumentation , 2020, KR.

[69]  Michael Kifer,et al.  Rulelog: Highly Expressive Semantic Rules with Scalable Deep Reasoning , 2017, RuleML+RR.

[70]  Eric D. Ragan,et al.  A Multidisciplinary Survey and Framework for Design and Evaluation of Explainable AI Systems , 2018, ACM Trans. Interact. Intell. Syst..

[71]  Henry Prakken,et al.  A two-phase method for extracting explanatory arguments from Bayesian networks , 2017, Int. J. Approx. Reason..

[72]  Moshe Y. Vardi,et al.  Graph Neural Networks Meet Neural-Symbolic Computing: A Survey and Perspective , 2020, IJCAI.

[73]  B. Levine.  Causal models. , 2009, Epidemiology.

[74]  D. Walton.  A Dialogue System for Evaluating Explanations , 2016 .

[75]  Francesca Toni,et al.  Argumentation for Explainable Scheduling , 2019, AAAI.

[76]  Eugene C. Freuder,et al.  Inference-Based Constraint Satisfaction Supports Explanation , 1996, AAAI/IAAI, Vol. 1.

[77]  Paolo Mancarella,et al.  Abductive Logic Programming , 1992, LPNMR.

[78]  Carlos Guestrin,et al.  "Why Should I Trust You?": Explaining the Predictions of Any Classifier , 2016, ArXiv.

[79]  Alan Fern,et al.  Explainable Reinforcement Learning via Reward Decomposition , 2019 .

[80]  Ulrich Endriss,et al.  Automated Justification of Collective Decisions via Constraint Solving , 2020, AAMAS.

[81]  Francesco Ricca,et al.  Debugging Non-ground ASP Programs: Technique and Graphical Tools , 2018, Theory and Practice of Logic Programming.

[82]  Phan Minh Dung,et al.  Assumption-Based Argumentation , 2009, Argumentation in Artificial Intelligence.

[83]  Jeehoon Kang,et al.  Promising-ARM/RISC-V: a simpler and faster operational concurrency model , 2019, PLDI.

[84]  David Isele,et al.  Safe Reinforcement Learning on Autonomous Vehicles , 2018, 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS).

[85]  Guillermo Ricardo Simari,et al.  Explanations, belief revision and defeasible reasoning , 2002, Artif. Intell..

[86]  Andrea Omicini,et al.  Towards XMAS: eXplainability through Multi-Agent Systems , 2019, AI&IoT@AI*IA.

[87]  Sameer Singh,et al.  Fooling LIME and SHAP: Adversarial Attacks on Post hoc Explanation Methods , 2020, AIES.

[88]  Yu Zhang,et al.  Plan Explanations as Model Reconciliation: Moving Beyond Explanation as Soliloquy , 2017, IJCAI.

[89]  Tim Miller,et al.  Explainable Reinforcement Learning Through a Causal Lens , 2019, AAAI.

[90]  Carlos José Pereira de Lucena,et al.  Pattern-based Explanation for Automated Decisions , 2014, ECAI.

[91]  Marco Maggini,et al.  A Constraint-Based Approach to Learning and Explanation , 2020, AAAI.

[92]  Francesca Toni,et al.  On Computing Explanations for Non-Acceptable Arguments , 2015 .

[93]  Francesca Toni,et al.  Justifying answer sets using argumentation , 2016, Theory Pract. Log. Program..

[94]  Joachim Diederich,et al.  Survey and critique of techniques for extracting rules from trained artificial neural networks , 1995, Knowl. Based Syst..

[95]  Zeynep Gozen Saribatur,et al.  Abstraction for ASP Planning , 2020, ECAI.

[96]  Johanna D. Moore,et al.  Explainable (and Maintainable) Expert Systems , 1985, IJCAI.

[97]  Xudong Luo,et al.  Explaining Best Decisions via Argumentation , 2014, ECSI.

[98]  Guillermo Ricardo Simari,et al.  Argument Theory Change: Revision Upon Warrant , 2008, COMMA.

[99]  LacaveCarmen,et al.  A review of explanation methods for Bayesian networks , 2002 .

[100]  Simon Parsons,et al.  Towards an argumentation-based approach to explainable planning , 2019 .

[101]  Dympna O'Sullivan,et al.  The Role of Explanations on Trust and Reliance in Clinical Decision Support Systems , 2015, 2015 International Conference on Healthcare Informatics.

[102]  Hector Geffner,et al.  Model-free, Model-based, and General Intelligence , 2018, IJCAI.

[103]  Peter J. Stuckey,et al.  Explaining alldifferent , 2012, ACSC.

[104]  Roberto Micalizio,et al.  Temporal Multiagent Plan Execution: Explaining What Happened , 2019, EXTRAAMAS@AAMAS.

[105]  Amina Adadi,et al.  Peeking Inside the Black-Box: A Survey on Explainable Artificial Intelligence (XAI) , 2018, IEEE Access.

[106]  Mathieu Serrurier,et al.  Agents that argue and explain classifications , 2007, Autonomous Agents and Multi-Agent Systems.

[107]  Anind K. Dey,et al.  Toolkit to support intelligibility in context-aware applications , 2010, UbiComp.

[108]  Zachary Chase Lipton.  The mythos of model interpretability , 2016, ACM Queue.

[109]  Leopoldo Bertossi.  An ASP-Based Approach to Counterfactual Explanations for Classification , 2020, RuleML+RR.

[110]  François Goasdoué,et al.  Computing and Explaining Query Answers over Inconsistent DL-Lite Knowledge Bases , 2019, J. Artif. Intell. Res..

[111]  Francesca Rossi,et al.  Computing Explanations and Implications in Preference-Based Configurators , 2002, International Workshop on Constraint Solving and Constraint Logic Programming.

[112]  Andrea Omicini,et al.  Interpretable Narrative Explanation for ML Predictors with LP: A Case Study for XAI , 2019, WOA.

[113]  V. S. Costa,et al.  Theory and Practice of Logic Programming , 2010 .

[114]  Floris Bex,et al.  Combining explanation and argumentation in dialogue , 2016, Argument Comput..

[115]  Ringo Baumann,et al.  If Nothing Is Accepted - Repairing Argumentation Frameworks , 2018, KR.

[116]  Floris Bex.  An integrated theory of causal stories and evidential arguments , 2015, ICAIL.

[117]  Qian Yang,et al.  Designing Theory-Driven User-Centric Explainable AI , 2019, CHI.

[118]  Raymond Reiter,et al.  A Theory of Diagnosis from First Principles , 1986, Artif. Intell..

[119]  Amanda Prorok,et al.  Culture-Based Explainable Human-Agent Deconfliction , 2020, AAMAS.

[120]  Geoff Sutcliffe,et al.  Evaluating general purpose automated theorem proving systems , 2001, Artif. Intell..

[121]  Catholijn M. Jonker,et al.  New Foundations of Ethical Multiagent Systems , 2020, AAMAS.

[122]  Daniele Theseider Dupré,et al.  Exploiting abstractions in cost-sensitive abductive problem solving with observations and actions , 2014, AI Commun..

[123]  Albert Cabellos-Aparicio,et al.  RouteNet: Leveraging Graph Neural Networks for Network Modeling and Optimization in SDN , 2020, IEEE Journal on Selected Areas in Communications.

[124]  Katsumi Nitta,et al.  Computing Abductive Argumentation in Answer Set Programming , 2009, ArgMAS.

[125]  William J. Clancey,et al.  Explanation in Human-AI Systems: A Literature Meta-Review, Synopsis of Key Ideas and Publications, and Bibliography for Explainable AI , 2019, ArXiv.

[126]  Francesca Toni,et al.  On Computing Explanations in Argumentation , 2015, AAAI.

[127]  H. Chad Lane,et al.  Building Explainable Artificial Intelligence Systems , 2006, AAAI.

[128]  Bei Shui Liao,et al.  Explanation Semantics for Abstract Argumentation , 2020, COMMA.

[129]  Francesca Toni,et al.  Abstract Argumentation for Case-Based Reasoning , 2016, KR.

[130]  Amos Azaria,et al.  AI for Explaining Decisions in Multi-Agent Environments , 2020, AAAI.

[131]  John F. Sowa,et al.  Knowledge Representation and Reasoning , 2000 .

[132]  Tran Cao Son,et al.  Explainable Planning Using Answer Set Programming , 2020, KR.

[133]  Alun D. Preece,et al.  Asking 'Why' in AI: Explainability of intelligent systems - perspectives and challenges , 2018, Intell. Syst. Account. Finance Manag..

[134]  Anthony Constantinou,et al.  Things to know about Bayesian networks: Decisions under uncertainty, part 2 , 2018 .

[135]  Bijan Parsia,et al.  Explaining Inconsistencies in OWL Ontologies , 2009, SUM.

[136]  Douglas Walton.  Explanations and Arguments Based on Practical Reasoning , 2009, ExaCt.

[137]  Subbarao Kambhampati,et al.  Hierarchical Expertise Level Modeling for User Specific Contrastive Explanations , 2018, IJCAI.

[138]  Joao Marques-Silva,et al.  On Relating Explanations and Adversarial Examples , 2019, NeurIPS.

[139]  John Fox,et al.  Cognitive systems at the point of care: The CREDO program , 2017, J. Biomed. Informatics.

[140]  Trevor J. M. Bench-Capon,et al.  Argumentation in artificial intelligence , 2007, Artif. Intell..

[141]  Edward Grefenstette,et al.  Differentiable Reasoning on Large Knowledge Bases and Natural Language , 2019, Knowledge Graphs for eXplainable Artificial Intelligence.

[142]  Abdallah Arioua,et al.  Query Answering Explanation in Inconsistent Datalog +/- Knowledge Bases , 2015, DEXA.

[143]  Claudette Cayrol,et al.  On bipolarity in argumentation frameworks , 2008, NMR.

[144]  Michael Fink,et al.  Under Consideration for Publication in Theory and Practice of Logic Programming Causal Graph Justifications of Logic Programs * , 2022 .

[145]  Senka Krivic,et al.  Towards Explainable AI Planning as a Service , 2019, ArXiv.

[146]  Mireia Ribera,et al.  Can we do better explanations? A proposal of user-centered explainable AI , 2019, IUI Workshops.

[147]  Francesca Toni,et al.  Explanation for Case-Based Reasoning via Abstract Argumentation , 2016, COMMA.

[148]  Enrico Pontelli,et al.  Under Consideration for Publication in Theory and Practice of Logic Programming Justifications for Logic Programs under Answer Set Semantics , 2022 .

[149]  Francesca Toni,et al.  Complexity Results and Algorithms for Bipolar Argumentation , 2019, AAMAS.

[150]  Johanna D. Moore,et al.  Explanations in knowledge systems: design for explainable expert systems , 1991, IEEE Expert.

[151]  Hiroshi Yamakawa,et al.  Autonomous Self-Explanation of Behavior for Interactive Reinforcement Learning Agents , 2017, HAI.

[152]  Mariela Morveli Espinoza,et al.  Towards an Explainable Argumentation-based Agent , 2020, STAIRS@ECAI.

[153]  Henry Prakken,et al.  Introduction to structured argumentation , 2014, Argument Comput..

[154]  Ruth M. J. Byrne,et al.  Counterfactuals in Explainable Artificial Intelligence (XAI): Evidence from Human Reasoning , 2019, IJCAI.

[155]  Michael Lawrence,et al.  The effects of structural characteristics of explanations on use of a DSS , 2006, Decis. Support Syst..

[156]  David Sarne,et al.  Summarizing agent strategies , 2019, Autonomous Agents and Multi-Agent Systems.

[157]  Lucia Specia,et al.  Human-in-the-loop Debugging Deep Text Classifiers , 2020, EMNLP.

[158]  Guillermo Ricardo Simari,et al.  Defeasible logic programming: DeLP-servers, contextual queries, and explanations for answers , 2014, Argument Comput..

[159]  Hilary Johnson,et al.  Explanation facilities and interactive systems , 1993, IUI '93.

[160]  Pierre Marquis,et al.  Extending abduction from propositional to first-order logic , 1991, FAIR.

[161]  David A. Rosenblueth,et al.  Learning Models from Temporal-Logic Properties via Explanations , 2007, ExaCt.

[162]  Adnan Darwiche,et al.  On The Reasons Behind Decisions , 2020, ECAI.

[163]  Pietro Baroni,et al.  From fine-grained properties to broad principles for gradual argumentation: A principled spectrum , 2019, Int. J. Approx. Reason..

[164]  David Danks,et al.  Different "Intelligibility" for Different Folks , 2020, AIES.

[165]  Tim Miller,et al.  A Grounded Interaction Protocol for Explainable Artificial Intelligence , 2019, AAMAS.

[166]  Tran Cao Son,et al.  Conditional Updates of Answer Set Programming and Its Application in Explainable Planning , 2020, AAMAS.

[167]  Jorge Fandinno,et al.  Answering the “why” in answer set programming – A survey of explanation approaches , 2018, Theory and Practice of Logic Programming.

[168]  Calin Belta,et al.  A Policy Search Method For Temporal Logic Specified Reinforcement Learning Tasks , 2017, 2018 Annual American Control Conference (ACC).

[169]  Douglas Walton,et al.  A dialogue system specification for explanation , 2011, Synthese.

[170]  Dov M. Gabbay,et al.  Abduction and Dialogical Proof in Argumentation and Logic Programming , 2014, ECAI.

[171]  Pierre Marquis,et al.  Consistency restoration and explanations in dynamic CSPs Application to configuration , 2002, Artif. Intell..

[172]  Jürg Kohlas,et al.  Probabilistic Argumentation Systems and Abduction , 2002, Annals of Mathematics and Artificial Intelligence.

[173]  Joao Marques-Silva,et al.  Abduction-Based Explanations for Machine Learning Models , 2018, AAAI.

[174]  Joe COLLENETTE,et al.  An Explainable Approach to Deducing Outcomes in European Court of Human Rights Cases Using ADFs , 2020, COMMA.

[175]  Luc De Raedt,et al.  Inductive Logic Programming: Theory and Methods , 1994, J. Log. Program..

[176]  Pietro Baroni,et al.  Relation-Based Counterfactual Explanations for Bayesian Network Classifiers , 2020, IJCAI.

[177]  Francesca Toni,et al.  AI-assisted Schedule Explainer for Nurse Rostering , 2020, AAMAS.

[178]  Subbarao Kambhampati,et al.  Why Can't You Do That HAL? Explaining Unsolvability of Planning Tasks , 2019, IJCAI.

[179]  Fahri Yetim,et al.  Collaborative Construction and Structuring of Explanations in Knowledge-Based Systems: A Discursive Framework , 2005, ExaCt.

[180]  Xavier Leroy,et al.  Formal C Semantics: CompCert and the C Standard , 2014, ITP.

[181]  Christophe Labreuche,et al.  Explaining Multi-Criteria Decision Aiding Models with an Extended Shapley Value , 2018, IJCAI.

[182]  André Elisseeff,et al.  Explanation Trees for Causal Bayesian Networks , 2008, UAI.

[183]  Avi Rosenfeld,et al.  Explainability in human–agent systems , 2019, Autonomous Agents and Multi-Agent Systems.

[184]  Francesca Toni,et al.  Extracting Dialogical Explanations for Review Aggregations with Argumentative Dialogical Agents , 2019, AAMAS.

[185]  J. Hoffmann,et al.  Plan-Space Explanation via Plan-Property Dependencies: Faster Algorithms & More Powerful Properties , 2020, IJCAI.

[186]  Hilary Putnam,et al.  A Computing Procedure for Quantification Theory , 1960, JACM.

[187]  Martin Lauer,et al.  An Algorithm for Distributed Reinforcement Learning in Cooperative Multi-Agent Systems , 2000, ICML.

[188]  L. Bottou.  From machine learning to machine reasoning , 2011, Machine Learning.

[189]  Ulrich Junker,et al.  QUICKXPLAIN: Preferred Explanations and Relaxations for Over-Constrained Problems , 2004, AAAI.

[190]  Barry O'Sullivan,et al.  Representative Explanations for Over-Constrained Problems , 2007, AAAI.

[191]  Mohan S. Kankanhalli,et al.  Trends and Trajectories for Explainable, Accountable and Intelligible Systems: An HCI Research Agenda , 2018, CHI.

[192]  Georg Gottlob,et al.  The complexity of logic-based abduction , 1993, JACM.

[193]  Serena Villata,et al.  On the Input/Output behavior of argumentation frameworks , 2014, Artif. Intell..

[194]  Jaime S. Cardoso,et al.  Machine Learning Interpretability: A Survey on Methods and Metrics , 2019, Electronics.

[195]  Dimos V. Dimarogonas,et al.  Scalable time-constrained planning of multi-robot systems , 2020, Auton. Robots.

[196]  Chris Russell,et al.  Counterfactual Explanations Without Opening the Black Box: Automated Decisions and the GDPR , 2017, ArXiv.

[197]  Johanna D. Moore,et al.  Pointing: A Way Toward Explanation Dialogue , 1990, AAAI.

[198]  Pat Langley,et al.  Explainable Agency for Intelligent Autonomous Systems , 2017, AAAI.

[199]  Paolo Mancarella,et al.  Computing ideal sceptical argumentation , 2007, Artif. Intell..

[200]  Marco Maggini,et al.  Human-Driven FOL Explanations of Deep Learning , 2020, IJCAI.

[201]  Sébastien Konieczny,et al.  Extension Enforcement in Abstract Argumentation as an Optimization Problem , 2015, IJCAI.

[202]  Marco Fiore,et al.  Network Slicing Meets Artificial Intelligence: An AI-Based Framework for Slice Management , 2020, IEEE Communications Magazine.

[203]  Joao Marques-Silva,et al.  Partial MUS Enumeration , 2013, AAAI.

[204]  H. Tompits,et al.  Catching the Ouroboros: On debugging non-ground answer-set programs , 2010, Theory and Practice of Logic Programming.

[205]  Carlos Uzcátegui,et al.  Preferences and explanations , 2003, Artif. Intell..

[206]  Francesca Toni,et al.  Data-Empowered Argumentation for Dialectically Explainable Predictions , 2020, ECAI.

[207]  Hans Tompits,et al.  A Meta-Programming Technique for Debugging Answer-Set Programs , 2008, AAAI.

[208]  Yike Guo,et al.  Explanations by arbitrated argumentative dispute , 2019, Expert Syst. Appl..

[209]  Rafael Peñaloza,et al.  Understanding the complexity of axiom pinpointing in lightweight description logics , 2017, Artif. Intell..

[210]  Léon Bottou,et al.  From machine learning to machine reasoning , 2011, Machine Learning.

[211]  Kostyantyn M. Shchekotykhin,et al.  Interactive Query-Based Debugging of ASP Programs , 2015, AAAI.

[212]  Extending Modular Semantics for Bipolar Weighted Argumentation , 2019, AAMAS.

[213]  Subbarao Kambhampati,et al.  Explicability? Legibility? Predictability? Transparency? Privacy? Security? The Emerging Landscape of Interpretable Agent Behavior , 2018, ICAPS.

[214]  Dov M. Gabbay,et al.  Handbook of the history of logic , 2004 .

[215]  Francisco Herrera,et al.  Explainable Artificial Intelligence (XAI): Concepts, Taxonomies, Opportunities and Challenges toward Responsible AI , 2020, Inf. Fusion.

[216]  Davide Calvaresi,et al.  Explainable Agents and Robots: Results from a Systematic Literature Review , 2019, AAMAS.

[217]  Chunyan Miao,et al.  Building More Explainable Artificial Intelligence With Argumentation , 2018, AAAI.

[218]  Grigoris Antoniou,et al.  Justifications for Logic Programming , 2013, LPNMR.

[219]  Henri Prade,et al.  Using arguments for making and explaining decisions , 2009, Artif. Intell..

[220]  David B. Leake.  Abduction, experience, and goals: a model of everyday abductive explanation , 1995, J. Exp. Theor. Artif. Intell..

[221]  Melvin Fitting,et al.  First-Order Logic and Automated Theorem Proving , 1990, Graduate Texts in Computer Science.

[222]  Marina De Vos,et al.  Answer Set Programming ? a Domain in Need of Explanation: A Position Paper , 2008, ExaCt.

[223]  Francesca Toni,et al.  A general framework for sound assumption-based argumentation dialogues , 2014, Artif. Intell..

[224]  Henry Prakken,et al.  Modelling Reasoning with Precedents in a Formal Dialogue Game , 2004, Artificial Intelligence and Law.

[225]  Joost Vennekens,et al.  Explaining Actual Causation in Terms of Possible Causal Processes , 2019, JELIA.

[226]  Phan Minh Dung,et al.  An Abstract, Argumentation-Theoretic Approach to Default Reasoning , 1997, Artif. Intell..

[227]  Elizabeth Sklar,et al.  A characterization of types of support between structured arguments and their relationship with support in abstract argumentation , 2018, Int. J. Approx. Reason..

[228]  Alexey Ignatiev,et al.  Towards Trustable Explainable AI , 2020, IJCAI.

[229]  Lars Kai Hansen,et al.  Interpretability in Intelligent Systems - A New Concept? , 2019, Explainable AI.

[230]  Thomas Lukasiewicz,et al.  Explanations for Query Answers under Existential Rules , 2019, IJCAI.

[231]  Wolfgang Faber.  Answer Set Programming , 2013, Reasoning Web.

[232]  Peter A. Flach,et al.  Conversational Explanations of Machine Learning Predictions Through Class-contrastive Counterfactual Statements , 2018, IJCAI.

[233]  Barry O'Sullivan,et al.  Generating Corrective Explanations for Interactive Constraint Satisfaction , 2005, CP.

[234]  Chunyan Miao,et al.  Context-based and Explainable Decision Making with Argumentation , 2018, AAMAS.

[235]  Christophe Labreuche,et al.  A general framework for explaining the results of a multi-attribute preference model , 2011, Artif. Intell..

[236]  Lalana Kagal,et al.  Explaining Explanations: An Overview of Interpretability of Machine Learning , 2018, 2018 IEEE 5th International Conference on Data Science and Advanced Analytics (DSAA).

[237]  Christophe Labreuche,et al.  Minimal and Complete Explanations for Critical Multi-attribute Decisions , 2011, ADT.

[238]  Robert A. Kowalski,et al.  Abduction Compared with Negation by Failure , 1989, ICLP.

[239]  Agnar Aamodt,et al.  Explanation in Case-Based Reasoning–Perspectives and Goals , 2005, Artificial Intelligence Review.

[240]  Xiuyi Fan.  On Generating Explainable Plans with Assumption-Based Argumentation , 2018, PRIMA.

[241]  David Poole,et al.  On the Comparison of Theories: Preferring the Most Specific Explanation , 1985, IJCAI.

[242]  Alun D. Preece,et al.  Interpretable to Whom? A Role-based Model for Analyzing Interpretable Machine Learning Systems , 2018, ArXiv.

[243]  Hector Geffner,et al.  Causal Theories for Nonmonotonic Reasoning , 1990, AAAI.

[244]  Subbarao Kambhampati,et al.  Plan Explanations as Model Reconciliation - An Empirical Study , 2018, ArXiv.

[245]  Chiaki Sakama.  Abduction in argumentation frameworks , 2018, J. Appl. Non Class. Logics.

[246]  Jiawei Han,et al.  CPAR: Classification based on Predictive Association Rules , 2003, SDM.

[247]  Bernard Moulin,et al.  Explanation and Argumentation Capabilities:Towards the Creation of More Persuasive Agents , 2002, Artificial Intelligence Review.

[248]  Dietmar Jannach,et al.  A systematic review and taxonomy of explanations in decision support and recommender systems , 2017, User Modeling and User-Adapted Interaction.

[249]  Ned Hall.  Structural equations and causation , 2007 .

[250]  Elizabeth Sklar,et al.  Explanation through Argumentation , 2018, HAI.

[251]  Freddy Lécué,et al.  On The Role of Knowledge Graphs in Explainable AI , 2020, PROFILES/SEMEX@ISWC.

[252]  Weng-Keen Wong,et al.  Principles of Explanatory Debugging to Personalize Interactive Machine Learning , 2015, IUI.

[253]  Adrian Weller,et al.  You Shouldn't Trust Me: Learning Models Which Conceal Unfairness From Multiple Explanation Methods , 2020, SafeAI@AAAI.

[254]  Claudette Cayrol,et al.  Bipolar abstract argumentation systems , 2009, Argumentation in Artificial Intelligence.

[255]  Gerhard Brewka,et al.  Preferred Subtheories: An Extended Logical Framework for Default Reasoning , 1989, IJCAI.

[256]  Maria Fox,et al.  Explainable Planning , 2017, ArXiv.

[257]  Christian Straßer,et al.  Abstract argumentation and explanation applied to scientific debates , 2011, Synthese.

[258]  Francesca Toni,et al.  Argumentation-Based Recommendations: Fantastic Explanations and How to Find Them , 2018, IJCAI.

[259]  Peter McBurney,et al.  Games That Agents Play: A Formal Framework for Dialogues between Autonomous Agents , 2002, J. Log. Lang. Inf..

[260]  Leila Amgoud,et al.  Evaluation of arguments in weighted bipolar graphs , 2018, Int. J. Approx. Reason..

[261]  Zohreh Shams,et al.  MARLeME: A Multi-Agent Reinforcement Learning Model Extraction Library , 2020, 2020 International Joint Conference on Neural Networks (IJCNN).

[262]  Judea Pearl,et al.  The seven tools of causal inference, with reflections on machine learning , 2019, Commun. ACM.

[263]  Tias Guns,et al.  Step-Wise Explanations of Constraint Satisfaction Problems , 2020, ECAI.

[264]  Eugene C. Freuder.  Explaining Ourselves: Human-Aware Constraint Reasoning , 2017, AAAI.

[265]  Pietro Baroni,et al.  On principle-based evaluation of extension-based argumentation semantics , 2007, Artif. Intell..

[266]  Guillermo Ricardo Simari,et al.  Formalizing dialectical explanation support for argument-based reasoning in knowledge-based systems , 2013, Expert Syst. Appl..

[267]  M. Ganesalingam,et al.  A Fully Automatic Theorem Prover with Human-Style Output , 2016, Journal of Automated Reasoning.