An evaluation of adaptive filtering in the context of realistic task-based information exploration

Exploratory search increasingly becomes an important research topic. Our interests focus on task-based information exploration, a specific type of exploratory search performed by a range of professional users, such as intelligence analysts. In this paper, we present an evaluation framework designed specifically for assessing and comparing performance of innovative information access tools created to support the work of intelligence analysts in the context of task-based information exploration. The motivation for the development of this framework came from our needs for testing systems in task-based information exploration, which cannot be satisfied by existing frameworks. The new framework is closely tied with the kind of tasks that intelligence analysts perform: complex, dynamic, and multiple facets and multiple stages. It views the user rather than the information system as the center of the evaluation, and examines how well users are served by the systems in their tasks. The evaluation framework examines the support of the systems at users' major information access stages, such as information foraging and sense-making. The framework is accompanied by a reference test collection that has 18 tasks scenarios and corresponding passage-level ground truth annotations. To demonstrate the usage of the framework and the reference test collection, we present a specific evaluation study on CAFE, an adaptive filtering engine designed for supporting task-based information exploration. This study is a successful use case of the framework, and the study indeed revealed various aspects of the information systems and their roles in supporting task-based information exploration.

[1]  Wai-Tat Fu,et al.  SNIF-ACT: A Model of Information Foraging on the World Wide Web , 2003, User Modeling.

[2]  James Allan,et al.  Topic detection and tracking: event-based information organization , 2002 .

[3]  Fredric C. Gey,et al.  Accessing Multilingual Information Repositories (6th Workshop of the Cross-Language Evaluation Forum, CLEF 2005) , 2006 .

[4]  Daqing He,et al.  HARD Experiment at Maryland: From Need Negotiation to Automated HARD Process , 2003, TREC.

[5]  Tefko Saracevic,et al.  Evaluation of evaluation in information retrieval , 1995, SIGIR '95.

[6]  Pia Borlund,et al.  The IIR evaluation model: a framework for evaluation of interactive information retrieval systems , 2003, Inf. Res..

[7]  N. Fuhr.  PAN-Uncovering Plagiarism , Authorship , and Social Software Misuse ImageCLEF 2013-Cross Language Image Annotation and Retrieval INEX-INitiative for the Evaluation of XML retrieval , 2002 .

[8]  Peter Ingwersen,et al.  The development of a method for the evaluation of interactive information retrieval systems , 1997, J. Documentation.

[9]  Cyril W. Cleverdon,et al.  Aslib Cranfield research project - Factors determining the performance of indexing systems; Volume 1, Design; Part 2, Appendices , 1966 .

[10]  Charles W. Krueger,et al.  New methods in software product line practice , 2006, CACM.

[11]  Michelle L. Gregory,et al.  From Question Answering to Visual Exploration , 2006, SIGIR 2006.

[12]  Peter Ingwersen,et al.  The Turn - Integration of Information Seeking and Retrieval in Context , 2005, The Kluwer International Series on Information Retrieval.

[13]  Pak Chung Wong,et al.  Have Green - A Visual Analytics Framework for Large Semantic Graphs , 2006, 2006 IEEE Symposium On Visual Analytics Science And Technology.

[14]  Stephen E. Robertson,et al.  The TREC 2002 Filtering Track Report , 2002, TREC.

[15]  Michelle X. Zhou,et al.  Interactive Visual Synthesis of Analytic Knowledge , 2006, 2006 IEEE Symposium On Visual Analytics Science And Technology.

[16]  Ellen M. Voorhees,et al.  TREC: Experiment and Evaluation in Information Retrieval (Digital Libraries and Electronic Publishing) , 2005 .

[17]  Gabriella Kazai.  Initiative for the Evaluation of XML Retrieval , 2009 .

[18]  Pablo Gervás,et al.  Personalisation in news delivery systems: Item summarization and multi-tier item selection using relevance feedback , 2005, Web Intell. Agent Syst..

[19]  Birger Larsen,et al.  The Interactive Track at INEX 2004 , 2004, INEX.

[20]  Ricardo Acosta-Diaz,et al.  An open source platform for Indexing and retrieval of multimedia information from a digital library of graduate thesis , 2006 .

[21]  Stephen E. Robertson,et al.  On the Evaluation of IR Systems , 1992, Inf. Process. Manag..

[22]  James Allan,et al.  HARD Track Overview in TREC 2003: High Accuracy Retrieval from Documents , 2003, TREC.

[23]  Gary Marchionini,et al.  Report on ACM SIGIR 2006 workshop on evaluating exploratory search systems , 2006, SIGF.

[24]  Yi Zhang.  Using bayesian priors to combine classifiers for adaptive filtering , 2004, SIGIR '04.

[25]  Cyril W. Cleverdon,et al.  Factors determining the performance of indexing systems , 1966 .

[26]  Peretz Shoval,et al.  Information Filtering: Overview of Issues, Research and Systems , 2001, User Modeling and User-Adapted Interaction.

[27]  Annika Waern,et al.  User Involvement in Automatic Filtering: An Experimental Study , 2004 .

[28]  Nicholas J. Belkin,et al.  The TREC Interactive Tracks: Putting the User into Search , 2005 .

[29]  Jaana Kekäläinen,et al.  The polyrepresentation continuum in IR , 2006, IIiX.

[30]  Gary Marchionini,et al.  Evaluating exploratory search systems: Introduction to special topic issue of information processing and management , 2008, Inf. Process. Manag..

[31]  Ryen W. White,et al.  Supporting exploratory search , 2006 .

[32]  José Luis Vicedo González,et al.  TREC: Experiment and evaluation in information retrieval , 2007, J. Assoc. Inf. Sci. Technol..

[33]  Ryen W. White,et al.  An implicit feedback approach for interactive information retrieval , 2006, Inf. Process. Manag..

[34]  Yiming Yang,et al.  Utility-based information distillation over temporally sequenced documents , 2007, SIGIR.

[35]  P. Pirolli,et al.  The Sensemaking Process and Leverage Points for Analyst Technology as Identified Through Cognitive Task Analysis , 2007 .

[36]  Gary Marchionini,et al.  Finding facts vs. browsing knowledge in hypertext systems , 1988, Computer.

[37]  Hsin-Hsi Chen,et al.  NTCIR workshop 6 meeting : proceedings of the 6th NTCIR workshop meeting on evaluation of information access technologies: information retrieval, question answering and cross-lingual information access , 2007 .

[38]  Gary Marchionini,et al.  Exploratory search , 2006, Commun. ACM.

[39]  Yiming Yang,et al.  Robustness of adaptive filtering methods in a cross-benchmark evaluation , 2005, SIGIR '05.

[40]  Thomas Ertl.  Guest Editor's Introduction: Special Section on the IEEE Symposium on Visual Analytics Science and Technology (VAST) , 2010, IEEE Trans. Vis. Comput. Graph..