Annotation Schemes to Encode Domain Knowledge in Medical Narratives

The broad goal of this study is to further the understanding of doctors' diagnostic styles and reasoning processes. We analyze and validate methods for annotating verbal diagnostic narratives collected together with eye-movement data. The long-term goal is to understand the cognitive reasoning and decision-making processes of medical experts, which could be useful for clinical information systems. The linguistic data set consists of transcribed recordings. Dermatologists were shown images of cutaneous conditions and asked to explain their observations aloud as they proceeded towards a diagnosis. We report on two linked annotation studies. In the first study, a subset of narratives were annotated by experts using a unique annotation scheme developed specifically for capturing decision-making components in the diagnostic process of dermatologists. We analyze annotator agreement as well as compare this annotation scheme to semantic types of the Unified Medical Language System as validation. In the second study, we explore the annotation of diagnostic correctness in the narratives at three relevant diagnostic steps, and we also explore the relationship between the two annotation schemes.