A Test Collection for Evaluating Legal Case Law Search

Test collection based evaluation represents the standard of evalua- tion for information retrieval systems. Legal IR, more speci cally case law retrieval, has no such standard test collection for evalua- tion. In this paper, we present a test collection for use in evaluating case law search, being the retrieval of judicial decisions relevant to a particular legal question. The collection is made available at ielab.io/caselaw.

[1]  Ioannis Anagnostopoulos,et al.  Multi-dimension Diversification in Legal Information Retrieval , 2016, WISE.

[2]  Minh-Tien Nguyen,et al.  Legal Question Answering using Ranking SVM and Deep Convolutional Neural Network , 2017, ArXiv.

[3]  Guido Zuccon,et al.  Automatic Query Generation from Legal Texts for Case Law Retrieval , 2017, AIRS.

[4]  Howard R. Turtle.  Natural language vs. Boolean query evaluation: a comparison of retrieval performance , 1994, SIGIR '94.

[5]  J. C. Smith,et al.  Beyond boolean search: FLEXICON, a legal tex-based intelligent system , 1991, ICAIL '91.

[6]  Ioannis Anagnostopoulos,et al.  Evaluation of Diversification Techniques for Legal Information Retrieval , 2017, Algorithms.

[7]  Guido Zuccon,et al.  Automatic Cited Decision Retrieval: Working notes of Ielab for FIRE Legal Track Precedence Retrieval Task , 2017, FIRE.

[8]  Guido Zuccon,et al.  A Test Collection for Evaluating Retrieval of Studies for Inclusion in Systematic Reviews , 2017, SIGIR.

[9]  Bhaskar Mitra,et al.  Neural Models for Information Retrieval , 2017, ArXiv.

[10]  Kevin Gerson Jd.  Evaluating Legal Information Retrieval Systems: How Do the Ranked-Retrieval Methods of WESTLAW and LEXIS Measure Up? , 1999 .

[11]  Arnab Bhattacharya,et al.  Overview of the FIRE 2017 IRLeD Track: Information Retrieval from Legal Documents , 2017, FIRE.

[12]  J. Fowler,et al.  Network Analysis and the Law: Measuring the Legal Importance of Precedents at the U.S. Supreme Court , 2007, Political Analysis.

[13]  Edwina L. Rissland,et al.  What You Saw Is What You Want: Using Cases to Seed Information Retrieval , 1997, ICCBR.

[14]  Catherine C. Marshall,et al.  A Human-Centered Framework for Ensuring Reliability on Crowdsourced Labeling Tasks , 2013, HCOMP.

[15]  Peter Bailey,et al.  UQV100: A Test Collection with Query Variability , 2016, SIGIR.

[16]  Robert A. Wilson,et al.  Computer Retrieval of Case Law , 1962 .

[17]  Anselmo Peñas,et al.  Overview of ResPubliQA 2009: Question Answering Evaluation over European Legislation , 2009, CLEF.

[18]  Marco Basaldella,et al.  Crowdsourcing Relevance Assessments: The Unexpected Benefits of Limiting the Time to Judge , 2016, HCOMP.

[19]  Colin Tapper.  Legal Information Retrieval by Computer : Applications and Implications , 2011 .