Evaluating Evaluations of Medical Diagnostic Systems

System evaluation in biomedical informatics should take place as an ongoing, strategically planned process, not as a single event or a small number of episodes. Complex software systems and accepted medical practices both evolve rapidly, so evaluators and readers of evaluations face moving targets. Thus, it is crucial for readers to be able to place any individual evaluation study into proper perspective. This advice applies to the nascent technology of medical diagnostic decision support systems (MDDSS). That the editor of a prestigious medical journal judged this entire technology’ based on a single, well-done, but intermediate-level and partial evaluation of several systems2 coupled with his own anecdotal experience emphasizes our professional obligation to characterize such systems and their evaluations responsibly