Evaluation of Mixed-Initiative Knowledge Base Development Methods and Tools

This paper presents a general framework for the empirical evaluation of mixed-initiative knowledge base development methods and tools. It also illustrates it with a knowledge acquisition experiment performed to validate a mixedinitiative method for acquiring problem solving rules directly from a subject matter expert. The main feature of this framework is that the evaluation is an integral part of system design and prototyping. The framework includes a repertoire of evaluation methods, guidelines for designing experiments based on these methods, and reusable evaluation utilities.