ILDC for CJPE: Indian Legal Documents Corpus for Court Judgment Prediction and Explanation

An automated system that could assist a judge in predicting the outcome of a case would help expedite the judicial process. For such a system to be practically useful, predictions by the system should be explainable. To promote research in developing such a system, we introduce ILDC (Indian Legal Documents Corpus). ILDC is a large corpus of 35k Indian Supreme Court cases annotated with original court decisions. A portion of the corpus (a separate test set) is annotated with gold standard explanations by legal experts. Based on ILDC, we propose the task of Court Judgment Prediction and Explanation (CJPE). The task requires an automated system to predict an explainable outcome of a case. We experiment with a battery of baseline models for case predictions and propose a hierarchical occlusion based model for explainability. Our best prediction model has an accuracy of 78% versus 94% for human legal experts, pointing towards the complexity of the prediction task. The analysis of explanations by the proposed algorithm reveals a significant difference in the point of view of the algorithm and legal experts for explaining the judgments, pointing towards scope for future research.

[1]  Kripabandhu Ghosh,et al.  A Comparative Study of Summarization Algorithms Applied to Legal Case Judgments , 2019, ECIR.

[2]  Dongyan Zhao,et al.  Learning to Predict Charges for Criminal Cases with Legal Basis , 2017, EMNLP.

[3]  Yiming Yang,et al.  XLNet: Generalized Autoregressive Pretraining for Language Understanding , 2019, NeurIPS.

[4]  Yoon Kim,et al.  Convolutional Neural Networks for Sentence Classification , 2014, EMNLP.

[5]  Lukasz Kaiser,et al.  Attention is All you Need , 2017, NIPS.

[6]  Carlos Guestrin,et al.  Anchors: High-Precision Model-Agnostic Explanations , 2018, AAAI.

[7]  Iryna Gurevych,et al.  A Web-based Tool for the Integrated Annotation of Semantic and Syntactic Structures , 2016, LT4DH@COLING.

[8]  Kripabandhu Ghosh,et al.  Identification of Rhetorical Roles of Sentences in Indian Legal Judgments , 2019, JURIX.

[9]  Zhiyuan Liu,et al.  Legal Judgment Prediction via Topological Learning , 2018, EMNLP.

[10]  Gaël Varoquaux,et al.  Scikit-learn: Machine Learning in Python , 2011, J. Mach. Learn. Res..

[11]  Minh Le Nguyen,et al.  Building Legal Case Retrieval Systems with Lexical Matching and Summarization using A Pre-Trained Phrase Scoring Model , 2019, ICAIL.

[12]  Ion Androutsopoulos,et al.  Neural Legal Judgment Prediction in English , 2019, ACL.

[13]  Josef van Genabith,et al.  Predicting the Law Area and Decisions of French Supreme Court Cases , 2017, RANLP.

[14]  Rob Fergus,et al.  Visualizing and Understanding Convolutional Networks , 2013, ECCV.

[15]  Ashutosh Modi,et al.  Topic Spotting using Hierarchical Networks with Self Attention , 2019, NAACL.

[16]  Daniel Jurafsky,et al.  Understanding Neural Networks through Representation Erasure , 2016, ArXiv.

[17]  Marcel van Gerven,et al.  Explainable Deep Learning: A Field Guide for the Uninitiated , 2020, J. Artif. Intell. Res..

[18]  D. Katz,et al.  A general approach for predicting the behavior of the Supreme Court of the United States , 2016, PloS one.

[19]  Matteo Pagliardini,et al.  Unsupervised Learning of Sentence Embeddings Using Compositional n-Gram Features , 2017, NAACL.

[20]  Ion Androutsopoulos,et al.  LEGAL-BERT: “Preparing the Muppets for Court’” , 2020, FINDINGS.

[21]  Nikolaos Aletras,et al.  Predicting judicial decisions of the European Court of Human Rights: a Natural Language Processing perspective , 2016, PeerJ Comput. Sci..

[22]  Thomas Wolf,et al.  DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter , 2019, ArXiv.

[23]  Pengfei Wang,et al.  Hierarchical Matching Network for Crime Classification , 2019, SIGIR.

[24]  Lukasz Kaiser,et al.  Reformer: The Efficient Transformer , 2020, ICLR.

[25]  Lysandre Debut,et al.  HuggingFace's Transformers: State-of-the-art Natural Language Processing , 2019, ArXiv.

[26]  Quoc V. Le,et al.  Distributed Representations of Sentences and Documents , 2014, ICML.

[27]  Alexander Binder,et al.  On Pixel-Wise Explanations for Non-Linear Classifier Decisions by Layer-Wise Relevance Propagation , 2015, PloS one.

[28]  Achim G. Hoffmann,et al.  Towards Automatic Generation of Catchphrases for Legal Case Reports , 2012, CICLing.

[29]  Weijia Jia,et al.  Legal Judgment Prediction via Multi-Perspective Bi-Feedback Network , 2019, IJCAI.

[30]  Omer Levy,et al.  RoBERTa: A Robustly Optimized BERT Pretraining Approach , 2019, ArXiv.

[31]  Byron C. Wallace,et al.  Attention is not Explanation , 2019, NAACL.

[32]  Zhiyuan Liu,et al.  CAIL2018: A Large-Scale Legal Dataset for Judgment Prediction , 2018, ArXiv.

[33]  Alon Lavie,et al.  METEOR: An Automatic Metric for MT Evaluation with High Levels of Correlation with Human Judgments , 2007, WMT@ACL.

[34]  Kripabandhu Ghosh,et al.  Automatic Catchphrase Identification from Legal Court Case Documents , 2017, CIKM.

[35]  Carlos Guestrin,et al.  "Why Should I Trust You?": Explaining the Predictions of Any Classifier , 2016, ArXiv.

[36]  Ming-Wei Chang,et al.  BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding , 2019, NAACL.

[37]  Chin-Yew Lin,et al.  ROUGE: A Package for Automatic Evaluation of Summaries , 2004, ACL 2004.

[38]  Maosong Sun,et al.  Iteratively Questioning and Answering for Interpretable Legal Judgment Prediction , 2020, AAAI.

[39]  Ankur Taly,et al.  Axiomatic Attribution for Deep Networks , 2017, ICML.

[40]  Diyi Yang,et al.  Hierarchical Attention Networks for Document Classification , 2016, NAACL.

[41]  Xin Jiang,et al.  Interpretable Charge Predictions for Criminal Cases: Learning to Generate Court Views from Fact Descriptions , 2018, NAACL.

[42]  Arman Cohan,et al.  Longformer: The Long-Document Transformer , 2020, ArXiv.

[43]  Xiaoyan Wang,et al.  Distinguish Confusing Law Articles for Legal Judgment Prediction , 2020, ACL.

[44]  Zhiyuan Liu,et al.  Few-Shot Charge Prediction with Discriminative Legal Attributes , 2018, COLING.

[45]  Beatriz de la Iglesia,et al.  Legal Judgement Prediction for UK Courts , 2020, ICISS.

[46]  Avanti Shrikumar,et al.  Learning Important Features Through Propagating Activation Differences , 2017, ICML.

[47]  Steven Bird,et al.  NLTK: The Natural Language Toolkit , 2002, ACL.

[48]  J. Fleiss.  Measuring nominal scale agreement among many raters. , 1971 .

[49]  Jeffrey Pennington,et al.  GloVe: Global Vectors for Word Representation , 2014, EMNLP.

[50]  Zhiyuan Liu,et al.  Automatic Judgment Prediction via Legal Reading Comprehension , 2018, CCL.

[51]  Deng Cai,et al.  Charge-Based Prison Term Prediction with Deep Gating Network , 2019, EMNLP.

[52]  Khalid Al-Kofahi,et al.  Information extraction from case law and retrieval of prior cases , 2003, Artif. Intell..

[53]  Salim Roukos,et al.  Bleu: a Method for Automatic Evaluation of Machine Translation , 2002, ACL.

[54]  Xin Jiang,et al.  Interpretable Rationale Augmented Charge Prediction System , 2018, COLING.