Generating Reasonable Legal Text through the Combination of Language Modeling and Question Answering

Due to the improvement of Language Modeling, the emerging NLP assistant tools aiming for text generation can greatly reduce the human workload on writing documents. However, the generation of legal text faces greater challenges than ordinary texts because of its high requirement for keeping logic reasonable, which can not be guaranteed by Language Modeling right now. To generate reasonable legal documents, we propose a novel method CoLMQA, which (1) combines Language Modeling and Question Answering, (2) generates texts with slots by Language Modeling, and (3) fills the slots by our proposed Question Answering method named Transformer-based Key-Value Memory Networks. In CoLMQA, the slots represent the part of the text that needs to be highly constrained by logic, such as the name of the law and the number of the law article. And the Question Answering fills the slots in context with the help of Legal Knowledge Base to keep logic reasonable. The experiment verifies the quality of legal documents generated by CoLMQA, surpassing the documents generated by pure Language Modeling.

[1]  Deng Cai,et al.  Charge-Based Prison Term Prediction with Deep Gating Network , 2019, EMNLP.

[2]  Xin Jiang,et al.  Interpretable Charge Predictions for Criminal Cases: Learning to Generate Court Views from Fact Descriptions , 2018, NAACL.

[3]  Andrew Chou,et al.  Semantic Parsing on Freebase from Question-Answer Pairs , 2013, EMNLP.

[4]  Jason Weston,et al.  Key-Value Memory Networks for Directly Reading Documents , 2016, EMNLP.

[5]  Jason Weston,et al.  Memory Networks , 2014, ICLR.

[6]  Jason Weston,et al.  Large-scale Simple Question Answering with Memory Networks , 2015, ArXiv.

[7]  Dongyan Zhao,et al.  Learning to Predict Charges for Criminal Cases with Legal Basis , 2017, EMNLP.

[8]  Maosong Sun,et al.  JEC-QA: A Legal-Domain Question Answering Dataset , 2019, AAAI.

[9]  Jason Weston,et al.  Wizard of Wikipedia: Knowledge-Powered Conversational agents , 2018, ICLR.

[10]  Lukás Burget,et al.  Recurrent neural network based language model , 2010, INTERSPEECH.

[11]  Hao Zhou,et al.  Variational Template Machine for Data-to-Text Generation , 2020, ICLR.

[12]  P. Cochat,et al.  Et al , 2008, Archives de pediatrie : organe officiel de la Societe francaise de pediatrie.

[13]  Garrison W. Cottrell,et al.  Improving Neural Story Generation by Targeted Common Sense Grounding , 2019, EMNLP.

[14]  Kenton Lee,et al.  Giving BERT a Calculator: Finding Operations and Arguments with Reading Comprehension , 2019, EMNLP.

[15]  Rajarshi Das,et al.  Question Answering on Knowledge Bases and Text using Universal Schema and Memory Networks , 2017, ACL.

[16]  Jerome R. Bellegarda,et al.  Statistical language model adaptation: review and perspectives , 2004, Speech Commun..

[17]  M. V. Rossum,et al.  In Neural Computation , 2022 .

[18]  Minlie Huang,et al.  A Knowledge-Enhanced Pretraining Model for Commonsense Story Generation , 2020, TACL.

[19]  Jian Zhang,et al.  SQuAD: 100,000+ Questions for Machine Comprehension of Text , 2016, EMNLP.

[20]  Steven Schockaert,et al.  Inducing Relational Knowledge from BERT , 2019, AAAI.

[21]  Zhiyuan Liu,et al.  Legal Judgment Prediction via Topological Learning , 2018, EMNLP.

[22]  Sameer Singh,et al.  Barack’s Wife Hillary: Using Knowledge Graphs for Fact-Aware Language Modeling , 2019, ACL.

[23]  Marko Marković,et al.  Legal Document Management: An Integrated Approach , 2019, Proceedings of the International Scientific Conference - Sinteza 2019.

[24]  Karl Branting,et al.  Techniques for automated drafting of judicial documents , 1998, Int. J. Law Inf. Technol..

[25]  Yansong Feng,et al.  Enhancing Key-Value Memory Neural Networks for Knowledge Based Question Answering , 2019, NAACL.

[26]  Ming-Wei Chang,et al.  BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding , 2019, NAACL.

[27]  Zhiyuan Liu,et al.  Few-Shot Charge Prediction with Discriminative Legal Attributes , 2018, COLING.