A Framework for Explainable Text Classification in Legal Document Review

Companies regularly spend millions of dollars producing electronically-stored documents in legal matters. Over the past two decades, attorneys have been using a variety of technologies to conduct this exercise, and most recently, parties on both sides of the ‘legal aisle’ are accepting the use of machine learning techniques like text classification to cull massive volumes of data and to identify responsive documents for use in these matters. While text classification is regularly used to reduce the discovery costs in legal matters, text classification also faces a peculiar perception challenge: amongst lawyers, this technology is sometimes looked upon as a black box Put simply, very little information is provided for attorneys to understand why documents are classified as responsive. In recent years, a group of AI and Machine Learning researchers have been actively researching Explainable AI. In an explainable AI system, actions or decisions are human understandable. In legal ‘document review’ scenarios, a document can be identified as responsive, as long as one or more of the text snippets (small passages of text) in a document are deemed responsive. In these scenarios, if text classification can be used to locate these responsive snippets, then attorneys could easily evaluate the model’s document classification decision. When deployed with defined and explainable results, text classification can drastically enhance the overall quality and speed of the document review process by reducing the time it takes to review documents. Moreover, explainable predictive coding provides lawyers with greater confidence in the results of that supervised learning task. This paper describes a framework for explainable text classification as a valuable tool in legal services: for enhancing the quality and efficiency of legal document review and for assisting in locating responsive snippets within responsive documents. This framework has been implemented in our legal analytics product, which has been used in hundreds of legal matters. We also report our experimental results using the data from an actual legal matter that used this type of document review.

[1]  Christine D. Piatko,et al.  Using “Annotator Rationales” to Improve Machine Learning for Text Categorization , 2007, NAACL.

[2]  Jianping Zhang,et al.  Empirical evaluations of active learning strategies in legal document review , 2017, 2017 IEEE International Conference on Big Data (Big Data).

[3]  Nicholas M. Pace,et al.  Where the Money Goes: Understanding Litigant Expenditures for Producing Electronic Discovery , 2012 .

[4]  Ye Zhang,et al.  Rationale-Augmented Convolutional Neural Networks for Text Classification , 2016, EMNLP.

[5]  Xiaohua Hu,et al.  Drexel at TREC 2014 Federated Web Search Track , 2014, TREC.

[6]  Foster J. Provost,et al.  Explaining Data-Driven Document Classifications , 2013, MIS Q..

[7]  Bruce G. Buchanan,et al.  The MYCIN Experiments of the Stanford Heuristic Programming Project , 1985 .

[8]  Yoav Goldberg,et al.  Understanding Convolutional Neural Networks for Text Classification , 2018, BlackboxNLP@EMNLP.

[9]  Hang Su,et al.  Towards Interpretable Deep Neural Networks by Leveraging Adversarial Examples , 2017, ArXiv.

[10]  Carlos Guestrin,et al.  "Why Should I Trust You?": Explaining the Predictions of Any Classifier , 2016, ArXiv.

[11]  Jason Eisner,et al.  Modeling Annotators: A Generative Approach to Learning from Annotator Rationales , 2008, EMNLP.

[12]  Maura R. Grossman,et al.  TREC 2016 Total Recall Track Overview , 2016, TREC.

[13]  Jianping Zhang,et al.  Explainable Text Classification in Legal Document Review A Case Study of Explainable Predictive Coding , 2018, 2018 IEEE International Conference on Big Data (Big Data).

[14]  Jianping Zhang,et al.  Empirical evaluations of preprocessing parameters' impact on predictive coding's effectiveness , 2016, 2016 IEEE International Conference on Big Data (Big Data).

[15]  William Yang Wang,et al.  Towards Explainable NLP: A Generative Explanation Framework for Text Classification , 2018, ACL.

[16]  Regina Barzilay,et al.  Rationalizing Neural Predictions , 2016, EMNLP.