Multi-Criteria Evaluation of Information Retrieval Tools

We propose a generic methodology for the evaluation of Text Mining/Search and Information Retrieval tools based on their functional conformity to a predefined set of functional requirements prioritized by distinguishable user profiles. The methodology is worked out and applied within the context of a research project concerning the assessment of intelligent exploitation tools for unstructured information sources in the police domain. We present the general setting of our work, give an overview of our evaluation approach, and discuss our methodology for testing in greater detail. These kinds of evaluations are particularly useful for both (potential)purchasers of exploitation tools, given the high investments in time and money required in becoming proficient in their use, and developers who aim at producing better quality software products.

[1]  Alexander Schnabl,et al.  Development of Multi-Criteria Metrics for Evaluation of Data Mining Algorithms , 1997, KDD.

[2]  Jesus Mena,et al.  Investigative Data Mining for Security and Criminal Detection , 2002 .

[3]  Gang Wang,et al.  Crime data mining: a general framework and some examples , 2004, Computer.

[4]  Marie-Francine Moens,et al.  Assessing the State of the Art of Commercial Tools for Unstructured Information Exploitation , 2006, BIS.

[5]  Jing Zhang,et al.  COPLINK Knowledge Management for Law Enforcement: Text Analysis, Visualization and Collaboration , 2001 .

[6]  William S. Cooper,et al.  On selecting a measure of retrieval effectiveness , 1973, J. Am. Soc. Inf. Sci..

[7]  Hsinchun Chen,et al.  COPLINK Center: Information and Knowledge Management for Law Enforcement , 2004, DG.O.

[8]  Peter Ingwersen,et al.  Information Retrieval Interaction , 1992 .

[9]  Ken W. Collier,et al.  A methodology for evaluating and selecting data mining software , 1999, Proceedings of the 32nd Annual Hawaii International Conference on Systems Sciences. 1999. HICSS-32. Abstracts and CD-ROM of Full Papers.

[10]  Hsinchun Chen,et al.  COPLINK: managing law enforcement data and knowledge , 2003, CACM.

[11]  John Zeleznikow,et al.  Decision support systems for police: Lessons from the application of data mining techniques to “soft” forensic evidence , 2006, Artificial Intelligence and Law.

[12]  M. Butler.  Information Retrieval Systems Characteristics, Testing, and Evaluation , 1970 .