The role of rules and examples in the process of knowledge acquisition in direct classification tasks

Abstract Shells provide a means for experts to easily develop expert systems for their area of expertise. However, rule bases need to be complete and free of contradictions. A set of 30 subjects, unfamiliar with shells except for initial orientation and training, were asked to develop a system for their personal preferences for a decision problem. The results of these systems were analyzed, leading to a number of conclusions. First, three types of rules used by the subjects were identified. Cutoff rules reflect preemptive treatment of decision rules. Examples reflect an attempt to enumerate all combinations of decision factors. Compensatory rules reflect attempts to balance trade-offs among the relative performance of decision cases. The implications of using these three types of rules are evaluated. Subjects validated their systems on a test bank of 18 cases. Subject responses to the impact of these test cases were evaluated, revealing that they thought that the test cases yielded more complete systems. Posttest evaluation of their systems for completeness and consistency also revealed that the systems still included significant gaps in rules. We conclude that computer aids to assist experts need to include means to assure consistency and completeness of knowledge bases. Further, at least some compensatory rules should be included for those cases that involve trade-offs.