Evaluating CBR Systems Using Different Data Sources: A Case Study

The complexity and high construction cost of case bases make it very difficult, if not impossible, to evaluate a CBR system, especially a knowledge-intensive CBR system, using statistical evaluation methods on many case bases. In this paper, we propose an evaluation strategy, which uses both many simple case bases and a few complex case bases to evaluate a CBR system, and show how this strategy may satisfy different evaluation goals. The identified evaluation goals are classified into two categories: domain-independent and domain-dependent. For the evaluation goals in the first category, we apply the statistical evaluation method using many simple case bases (for example, UCI data sets); for evaluation goals in the second category, we apply different, relatively weak, evaluation methods on a few complex domain-specific case bases. We apply this combined evaluation strategy to evaluate our knowledge-intensive conversational CBR method as a case study.

[1]  Adele E. Howe,et al.  How evaluation guides AI research , 1988 .

[2]  Paul R. Cohen,et al.  Toward AI research methodology: three case studies in evaluation , 1989, IEEE Trans. Syst. Man Cybern..

[3]  David W. Aha,et al.  Generalizing from Case studies: A Case Study , 1992, ML.

[4]  Agnar Aamodt,et al.  Case-Based Reasoning: Foundational Issues, Methodological Variations, and System Approaches , 1994, AI Commun..

[5]  Ashwin Ram,et al.  Systematic Evaluation of Design Decisions in CBR Systems , 1994 .

[6]  Herbert A. Simon,et al.  Artificial Intelligence: An Empirical Science , 1995, Artif. Intell..

[7]  S.J.J. Smith,et al.  Empirical Methods for Artificial Intelligence , 1995 .

[8]  Barry Smyth,et al.  Advances in Case-Based Reasoning , 1996, Lecture Notes in Computer Science.

[9]  Luc Lamontagne,et al.  Case-Based Reasoning Research and Development , 1997, Lecture Notes in Computer Science.

[10]  Catherine Blake,et al.  UCI Repository of machine learning databases , 1998 .

[11]  Ian H. Witten,et al.  Data mining: practical machine learning tools and techniques, 3rd Edition , 1999 .

[12]  Michael Stonebraker,et al.  The Morgan Kaufmann Series in Data Management Systems , 1999 .

[13]  Padraig Cunningham,et al.  A Dynamic Approach to Reducing Dialog in On-Line Decision Guides , 2000, EWCBR.

[14]  D. Bridge,et al.  Partial Orders and Indifference Relations: Being Purposefully Vague in Case-Based Retrieval , 2000, EWCBR.

[15]  Cynthia A. Thompson,et al.  Personalized Conversational Case-Based Recommendation , 2000, EWCBR.

[16]  Enrico Blanzieri,et al.  Advances in Case-Based Reasoning: 5th European Workshop, EWCBR 2000 Trento, Italy, September 6-9, 2000 Proceedings , 2000 .

[17]  Pedro A. González-Calero,et al.  An Architecture for Knowledge Intensive CBR Systems , 2000, EWCBR.

[18]  Ian Witten,et al.  Data Mining , 2000 .

[19]  David McSherry.  Minimizing Dialog Length in Interactive Case-Based Reasoning , 2001, IJCAI.

[20]  Kalyan Moy Gupta,et al.  Exploiting Taxonomic and Causal Relations in Conversational Case Retrieval , 2002, ECCBR.

[21]  Bruce M. McLaren,et al.  Extensionally defining principles and cases in ethics: An AI model , 2003, Artif. Intell..

[22]  David McSherry,et al.  Interactive Case-Based Reasoning in Sequential Diagnosis , 2001, Applied Intelligence.

[23]  Steven Bogaerts,et al.  Facilitating CBR for Incompletely-Described Cases: Distance Metrics for Partial Problem Descriptions , 2004, ECCBR.

[24]  Qing He,et al.  Intelligent Information Processing II , 2004 .

[25]  Xin Tong,et al.  Component Retrieval Using Conversational Case-Based Reasoning , 2004, Intelligent Information Processing.

[26]  Agnar Aamodt,et al.  Knowledge-Intensive Case-Based Reasoning in CREEK , 2004, ECCBR.

[27]  David W. Aha,et al.  Conversational Case-Based Reasoning , 2004, Applied Intelligence.

[28]  Jing Wu,et al.  Enhancing the Effectiveness of Interactive Case-Based Reasoning with Clustering and Decision Forests , 2004, Applied Intelligence.

[29]  Xin Tong,et al.  Dynamic feature weighting in nearest neighbor classifiers , 2004, Proceedings of 2004 International Conference on Machine Learning and Cybernetics (IEEE Cat. No.04EX826).

[30]  Peter Funk,et al.  Advances in case-based reasoning : 7th European Conference, ECCBR 2004 Madrid, Spain, August 30 - September 2, 2004 : proceedings , 2004 .

[31]  Hideo Shimazu,et al.  ExpertClerk: A Conversational Case-Based Reasoning Tool forDeveloping Salesclerk Agents in E-Commerce Webshops , 2002, Artificial Intelligence Review.

[32]  David McSherry,et al.  Explanation in Recommender Systems , 2005, Artificial Intelligence Review.

[33]  Xin Tong,et al.  Comparing similarity calculation methods in conversational CBR , 2005, IRI -2005 IEEE International Conference on Information Reuse and Integration, Conf, 2005..

[34]  Barry Smyth,et al.  Explaining Compound Critiques , 2005, Artificial Intelligence Review.

[35]  Ian H. Witten,et al.  Data mining - practical machine learning tools and techniques, Second Edition , 2005, The Morgan Kaufmann series in data management systems.

[36]  Agnar Aamodt,et al.  A Knowledge-Intensive Method for Conversational CBR , 2005, ICCBR.

[37]  Agnar Aamodt,et al.  Explanation in Case-Based Reasoning–Perspectives and Goals , 2005, Artificial Intelligence Review.

[38]  Mingyang Gu,et al.  Component Retrieval Using Knowledge-Intensive Conversational CBR , 2006, IEA/AIE.

[39]  Agnar Aamodt,et al.  Dialog Learning in Conversational CBR , 2006, FLAIRS.

[40]  John Fulcher,et al.  Advances in Applied Artificial Intelligence , 2006 .

[41]  D. A V I,et al.  Advances in conversational case-based reasoning , 2006 .