Validation and Testing of Knowledge-Based Systems - How bad can it get?

My subtitle is not intended as a slighting reference to the current state of the art for validation and veriication of knowledge-based systems, but as a reminder that for many systems the important question concerning deployment is not \how well does it work?" but \how badly can it fail?" This is most obviously the case in safety-critical systems, but it can apply to any system where the consequences of certain types of failure may be incommensurate with the beneets of normal operation. For conventional software, it is a topic of active debate whether the techniques that are most eeective for ensuring that a system \works well" are also the most eeective for showing that it \cannot go badly wrong." At the core of this debate is the question whether safety should be distinguished from reliability. Veriication and validation for knowledge-based systems has been almost exclusively concerned with showing that these systems can work well. But if knowledge-based systems are to become accepted as components of larger systems that perform critical functions, it will also be necessary to consider the question of how badly they can go wrong. In this regard, it will be useful to consider relevant experience with conventional software and to introduce some of the terms and concepts from \de-pendable systems," \software reliability," and \software safety."