Evaluating visual analytics for text information retrieval

Retrieving information from document collections is necessary in many contexts, for example, researchers wish to retrieve papers on a research topic, physicians search for patient records related to a certain condition, police investigators seek for relationships in criminal reports. Common to these scenarios are users in need of identifying relevant textual information in a document collection. The task is challenging, especially when users hope for a retrieval process that misses none or very few of the relevant documents. Visual Analytics (VA) approaches are often advocated to support document retrieval tasks. VA relies on integrating interactive visualizations and machine learning algorithms so that a domain expert can gradually steer a system into identifying the relevant documents. As an example, TRIVIR is a state-of-the-art system that allows exploring a corpus while providing feedback to a classifier that suggests potentially relevant documents to a reference query document. Assessing VA-supported Information Retrieval (IR) strategies is also challenging, as using these systems typically involves many conceptual and practical aspects and text retrieval tasks can demand considerable cognitive effort. In this paper, we present results from observational studies on VA-supported text information retrieval. We conducted sessions with graduate students and researchers using TRIVIR to explore scientific papers for purposes of literature review. A first study allowed us to collect opinions and identify some usability issues and practical limitations of the available implementation. After handling some critical issues observed at the interface level, we conducted a second round of sessions in order to collect further user opinions regarding a retrieval process assisted with VA. We concluded that most users have a very positive view of the system's usability and its ability to facilitate their retrieval tasks. Nonetheless, we also learnt that a proper introduction to the role of the interface elements is important and that conveying the underlying conceptual model and its limitations can be difficult. We observed considerable variation in user assessment of the specific functionalities and some users may face practical difficulties in using the system autonomously in an optimal way.