Matching the Level of Evaluation to a project's Stage of Development

The first issue of the Journal of the American Medical lnformatics Association (JAMIA) included a paper that proposed a framework for design of applied medical informatics projects and the evaluative studies that are part of them. A fundamental concept was that innovation in medical informatics takes place through sequential stages of system development. The first stage is specification, followed by component development, combination of components into a system, integration of the system into the environment, and, finally, routine use. The level of evaluation should be tuned to the stage of development. For example, problem definition and bench testing are appropriate during the specification stage. Field and validity testing become possible as components are combined into a system. Evaluation of efficacy is not practical until the sytem is integrated into the environment.