Conversational vs Traditional: Comparing Search Behavior and Outcome in Legal Case Retrieval

In recent years, legal case retrieval has attracted much attention in the IR research community. It aims to retrieve supporting cases for a given query case and contributes to better legal systems. While using a legal case retrieval system, users always feel difficult to construct accurate queries to express their information need, especially when they lack sufficient domain knowledge. Since conversational search has been widely recognized to fulfill users' complex and exploratory information need, we investigate whether conversational search paradigm can be adopted to improve users' legal case retrieval experience. We design a laboratory-based study to collect users' interaction behaviors and explicit feedback signals while using traditional and agent-mediated conversational legal case retrieval systems. Based on the collected data, we compare search behavior and outcome of these two different kinds of interaction paradigms. Compared with the traditional one, experimental results show that users can achieve better retrieval performance with the conversational case retrieval system. Moreover, conversational system can also save users' efforts in formulating queries and examining results.

[1]  Howard R. Turtle.  Text retrieval in the legal world , 1995, Artificial Intelligence and Law.

[2]  J. Fleiss.  Measuring nominal scale agreement among many raters. , 1971 .

[3]  Ben Carterette,et al.  Overview of the TREC 2014 Session Track , 2014, TREC.

[4]  John O. McGinnis,et al.  The Great Disruption: How Machine Intelligence Will Transform the Role of Lawyers in the Delivery of Legal Services , 2014, Actual Problems of Economics and Law.

[5]  Nicholas J. Belkin,et al.  Cases, scripts, and information-seeking strategies: On the design of interactive information retrieval systems , 1995 .

[6]  Hanjo Hamann.  The German Federal Courts Dataset 1950–2019: From Paper Archives to Linked Open Data , 2019, Journal of Empirical Legal Studies.

[7]  Filip Radlinski,et al.  A Theoretical Framework for Conversational Search , 2017, CHIIR.

[8]  Ryen W. White,et al.  Struggling and Success in Web Search , 2015, CIKM.

[9]  Cristiana Santos,et al.  On the concept of relevance in legal information retrieval , 2017, Artificial Intelligence and Law.

[10]  Diane Kelly,et al.  Methods for Evaluating Interactive Information Retrieval Systems with Users , 2009, Found. Trends Inf. Retr..

[11]  Paul Solomon,et al.  Conversation in information-seeking contexts: A test of an analytical framework , 1997 .

[12]  Yiqun Liu,et al.  How Does Domain Expertise Affect Users’ Search Interaction and Outcome in Exploratory Search? , 2018, ACM Trans. Inf. Syst..

[13]  Xu Chen,et al.  Towards Conversational Search and Recommendation: System Ask, User Respond , 2018, CIKM.

[14]  Ryen W. White,et al.  Characterizing the influence of domain expertise on web search behavior , 2009, WSDM '09.

[15]  W. Bruce Croft,et al.  Asking Clarifying Questions in Open-Domain Information-Seeking Conversations , 2019, SIGIR.

[16]  Filip Radlinski,et al.  Common Conversational Community Prototype: Scholarly Conversational Assistant , 2020, ArXiv.

[17]  Hamed Zamani,et al.  Macaw: An Extensible Conversational Information Seeking Platform , 2019, SIGIR.

[18]  John O. McGinnis,et al.  Law’s Algorithm , 2014 .

[19]  Eugene Agichtein,et al.  Find it if you can: a game for modeling different types of web search success using interaction data , 2011, SIGIR.