Qualitative analysis of workflow modifications used to generate the reference standard for the 2010 i2b2/VA challenge.

The Department of Veterans Affairs (VA) and the Informatics for Integrating Biology and the Bedside (i2b2) team partnered to generate the reference standard for the 2010 i2b2/VA challenge task on concept extraction, assertion classification, and relation classification. The purpose of this paper is to report an in-depth qualitative analysis of the experience and perceptions of human annotators for these tasks. Transcripts of semi-structured interviews were analyzed using qualitative methods to identify key constructs and themes related to these annotation tasks. Interventions were embedded with these tasks using pre-annotation of clinical concepts and a modified annotation workflow. From the human perspective, annotation tasks involve an inherent conflict between bias, accuracy, and efficiency. This analysis deepens understanding of the biases, complexities and impact of variations in the annotation process that may affect annotation task reliability and reference standard validity that are generalizable for other similar large-scale clinical corpus annotation projects.

[1]  Arie W. Kruglanski,et al.  Lay Epistemics and Human Knowledge: Cognitive and Motivational Bases , 2013 .

[2]  Özlem Uzuner,et al.  Extracting medication information from clinical text , 2010, J. Am. Medical Informatics Assoc..

[3]  Bibb Latané,et al.  Demonstrating Dynamic Social Impact: Consolidation, Clustering, Correlation, and (Sometimes) the Correct Answer , 1998 .

[4]  Philip V. Ogren,et al.  Knowtator: A Protégé plug-in for annotated corpus construction , 2006, NAACL.

[5]  Fei Xia,et al.  Community annotation experiment for ground truth generation for the i2b2 medication challenge , 2010, J. Am. Medical Informatics Assoc..

[6]  E. Hollnagel.  The Etto Principle: Efficiency-Thoroughness Trade-Off: Why Things That Go Right Sometimes Go Wrong , 2009 .

[7]  Murray Turoff,et al.  The Delphi Method: Techniques and Applications , 1976 .

[8]  C. Sansone,et al.  Effects of instruction on intrinsic interest: the importance of context. , 1989, Journal of personality and social psychology.

[9]  Yuan Luo,et al.  Identifying patient smoking status from medical discharge records. , 2008, Journal of the American Medical Informatics Association : JAMIA.

[10]  Derek J. Koehler,et al.  Heuristics and Biases: The Calibration of Expert Judgment: Heuristics and Biases Beyond the Laboratory , 2002 .

[11]  Charlene R. Weir,et al.  Research Paper: A Cognitive Task Analysis of Information Management Strategies in a Computerized Provider Order Entry Environment , 2007, J. Am. Medical Informatics Assoc..

[12]  M. Patton.  Qualitative research and evaluation methods , 1980 .

[13]  S W Tu,et al.  PROTEGE-II: computer support for development of intelligent systems from libraries of components. , 1995, Medinfo. MEDINFO.

[14]  Angus Roberts,et al.  Building a semantically annotated corpus of clinical texts , 2009, J. Biomed. Informatics.

[15]  Ralph Grishman,et al.  Message Understanding Conference- 6: A Brief History , 1996, COLING.

[16]  R. Baumeister,et al.  The need to belong: desire for interpersonal attachments as a fundamental human motivation. , 1995, Psychological bulletin.

[17]  Jutta Heckhausen,et al.  Motivation and action , 1991 .

[18]  Özlem Uzuner,et al.  Viewpoint Paper: Recognizing Obesity and Comorbidities in Sparse Data , 2009, J. Am. Medical Informatics Assoc..

[19]  Shuying Shen,et al.  2010 i2b2/VA challenge on concepts, assertions, and relations in clinical text , 2011, J. Am. Medical Informatics Assoc..

[20]  Kentaro Fujita,et al.  Planning and the Implementation of Goals , 2004 .

[21]  William K. Estes,et al.  Classification and cognition , 1994 .

[22]  Robert S. Wyer,et al.  Social Comprehension and Judgment: The Role of Situation Models, Narratives, and Implicit Theories , 2003 .

[23]  William R. Hersh,et al.  Enhancing Access to the Bibliome: The TREC Genomics Track , 2004, MedInfo.

[24]  Angus Roberts,et al.  The CLEF Corpus: Semantic Annotation of Clinical Text , 2007, AMIA.

[25]  Peter Szolovits,et al.  Evaluating the state-of-the-art in automatic de-identification. , 2007, Journal of the American Medical Informatics Association : JAMIA.

[26]  Robert S. Baron,et al.  So Right It's Wrong: Groupthink and the Ubiquitous Nature of Polarized Group Decision Making , 2005 .

[27]  Dean F Sittig,et al.  Understanding Inter-rater Disagreement: A Mixed Methods Approach. , 2010, AMIA ... Annual Symposium proceedings. AMIA Symposium.

[28]  D. Middleton.  Cognition and communication at work: Talking work: Argument, common knowledge, and improvisation in teamwork , 1996 .

[29]  Alfonso Valencia,et al.  Overview of BioCreAtIvE: critical assessment of information extraction for biology , 2005, BMC Bioinformatics.