Experiments with the Negotiated Boolean Queries of the TREC 2007 Legal Discovery Track

We analyze the results of several experimental runs submitted for the TREC 2007 Legal Track (also sometimes known as the Legal Discovery Track). We submitted 4 boolean query runs (the initial proposal by the defendant, the rejoinder by the plaintiff, the final negotiated query, and a variation of the final query which had proximity distances doubled). We submitted 2 vector query runs (one based on the keywords of the final negotiated query, and another based on the (natural language) request text). We submitted a blind feedback run based on the final negotiated boolean query. Finally, we submitted a fusion run of the final boolean, request text and final vector runs. We found that none of the runs had a higher mean estimated Recall@B than the original final negotiated boolean query.

[1]  Douglas W. Oard,et al.  TREC 2006 Legal Track Overview , 2006, TREC.

[2]  Howard R. Turtle.  Natural language vs. Boolean query evaluation: a comparison of retrieval performance , 1994, SIGIR '94.

[3]  Stephen Tomlinson.  Enterprise, QA, Robust and Terabyte Experiments with Hummingbird SearchServer at TREC 2005 , 2005, TREC.

[4]  Bogdan Sacaleanu,et al.  Working Notes for the CLEF 2008 Workshop , 2008 .

[5]  Ellen M. Voorhees,et al.  Overview of the TREC 2004 Robust Retrieval Track , 2004 .

[6]  Gary Promhouse,et al.  Experiments with TREC using the Open Text Livelink Engine , 1996, TREC.

[7]  Ellen M. Voorhees,et al.  Retrieval evaluation with incomplete information , 2004, SIGIR '04.

[8]  Noriko Kando.  NII Test Collection for IR Home Page , 2001 .

[9]  Stephen Tomlinson.  CJK Experiments with Hummingbird SearchServerTM at NTCIR-4 , 2004, NTCIR.

[10]  David R. Karger,et al.  Less is More Probabilistic Models for Retrieving Fewer Relevant Documents , 2006 .

[11]  Stephen Tomlinson.  Early precision measures: implications from the downside of blind feedback , 2006, SIGIR '06.

[12]  Stephen Tomlinson.  European Ad Hoc Retrieval Experiments with Hummingbird SearchServerTM at CLEF 2005 , 2005, CLEF.

[13]  Djoerd Hiemstra,et al.  Retrieving Web Pages Using Content, Links, URLs and Anchors , 2001, TREC.

[14]  John D. Holt,et al.  Boolean System Revisited: Its Performance and its Behavior , 1995, TREC.

[15]  Emine Yilmaz,et al.  Estimating average precision with incomplete and imperfect judgments , 2006, CIKM '06.

[16]  Stephen E. Robertson,et al.  On GMAP: and other transformations , 2006, CIKM '06.

[17]  Stephen E. Robertson,et al.  Okapi at TREC-3 , 1994, TREC.

[18]  Stephen E. Robertson,et al.  GatfordCentre for Interactive Systems ResearchDepartment of Information , 1996 .

[19]  Stephen Tomlinson,et al.  Comparing the Robustness of Expansion Techniques and Retrieval Measures , 2006, CLEF.

[20]  Shlomo Argamon,et al.  Building a test collection for complex document information processing , 2006, SIGIR.