Evaluating Comparative Evaluation Models

Abstract In initially developing a model of case-based comparative evaluation, we havefollowed a plan that involved an early empirical evaluation by a noted domainexpert. We describe the empirical evaluation and the lessons learned after weintroduce the program and its knowledge sources and work through anextended example. TRUTH-TELLER, a program for testing a Case-BasedReasoning (CBR) knowledge representation in practical ethics, compares casespresenting ethical dilemmas about whether to tell the truth. Its comparisons listethically relevant similarities and differences (i.e., reasons for telling or not tellingthe truth which apply to both cases, and reasons which apply more strongly inone case than another or which apply only to one case). In generating context-sensitive comparisons, the program reasons about reasons which may invokeethical principles or selfish considerations. In a preliminary evaluation, aprofessional ethicist scored the program’s output for randomly-selected pairs ofcases