'Healthy' Coreference: Applying Coreference Resolution to the Health Education Domain

This thesis investigates coreference and its resolution within the domain of health education. Coreference is the relationship between two linguistic expressions that refer to the same real-world entity, and resolution involves identifying this relationship among sets of referring expressions. The coreference resolution task is considered among the most difficult of problems in Artificial Intelligence; in some cases, resolution is impossible even for humans. For example, she in the sentence Lynn called Jennifer while she was on vacation is genuinely ambiguous: the vacationer could be either Lynn or Jennifer. There are three primary motivations for this thesis. The first is that health education has never before been studied in this context. So far, the vast majority of coreference research has focused on news. Secondly, achieving domain-independent resolution is unlikely without understanding the extent to which coreference varies across different genres. Finally, coreference pervades language and is an essential part of coherent discourse. Its effective use is a key component of easy-tounderstand health education materials, where readability is paramount. No suitable corpus of health education materials existed, so our first step was to create one. The comprehensive analysis of this corpus, which required manual annotation of coreference, confirmed our hypothesis that the coreference used in health education differs substantially from that in previously studied domains. This analysis was then used to shape the design of a knowledge-lean algorithm for resolving coreference. This algorithm performed surprisingly well on this corpus, e.g., successfully resolving over 85% of all pronouns when evaluated on unseen data. Despite the importance of coreferentially annotated corpora, only a handful are known to exist, likely because of the difficulty and cost of reliably annotating coreference. The paucity of genres represented in these existing annotated corpora creates an implicit bias in domain-independent coreference resolution. In an effort to address these issues, we plan to make our health education corpus available to the wider research community, hopefully encouraging a broader focus in the future.

[1]  Simone Paolo Ponzetto,et al.  Exploiting Semantic Role Labeling, WordNet and Wikipedia for Coreference Resolution , 2006, NAACL.

[2]  E. J. Mayeaux,et al.  Reading ability of parents compared with reading level of pediatric patient education materials. , 1994, Pediatrics.

[3]  Sanda M. Harabagiu,et al.  COREFDRAW: a tool for annotation and visualization of coreference data , 2001, Proceedings 13th IEEE International Conference on Tools with Artificial Intelligence. ICTAI 2001.

[4]  Thomas S. Morton,et al.  WordFreak: An Open Tool for Linguistic Annotation , 2003, HLT-NAACL.

[5]  Ruslan Mitkov,et al.  Automatic Anaphora Resolution: Limits, Impediments, and Ways Forward , 2002, PorTAL.

[6]  Kalina Bontcheva,et al.  A Light-weight Approach to Coreference Resolution for Named Entities in Text , 2002 .

[7]  D. Marcu,et al.  Experiments in Constructing a Corpus of Discourse Trees : Problems , Annotation Choices , Issues , 1999 .

[8]  S. Siegel,et al.  Nonparametric Statistics for the Behavioral Sciences , 2022, The SAGE Encyclopedia of Research Design.

[9]  M. Gernsbacher,et al.  Accessing Sentence Participants: The Advantage of First Mention. , 1988, Journal of memory and language.

[10]  Nancy A. Chinchor,et al.  Overview of MUC-7 , 1998, MUC.

[11]  R. P. Fishburne,et al.  Derivation of New Readability Formulas (Automated Readability Index, Fog Count and Flesch Reading Ease Formula) for Navy Enlisted Personnel , 1975 .

[12]  Renata Vieira,et al.  An Empirically-based System for Processing Definite Descriptions , 2000, CL.

[13]  R. L. Bradshaw,et al.  RESULTS AND ANALYSIS. , 1971 .

[14]  Yorick Wilks,et al.  University of Sheffield: Description of the LaSIE System as Used for MUC-6 , 1995, MUC.

[15]  R. Flesch A new readability yardstick. , 1948, The Journal of applied psychology.

[16]  Barbara Di Eugenio,et al.  On the Usage of Kappa to Evaluate Agreement on Coding Tasks , 2000, LREC.

[17]  Olga Uryupina,et al.  Knowledge acquisition for coreference resolution , 2007 .

[18]  Breck Baldwin,et al.  University of Pennsylvania: description of the University of Pennsylvania system used for MUC-6 , 1995, MUC.

[19]  Renata Vieira,et al.  Definite Description Processing in Unrestricted Text , 1998 .

[20]  Helmut Schmidt,et al.  Probabilistic part-of-speech tagging using decision trees , 1994 .

[21]  Klaus Krippendorff,et al.  Content Analysis: An Introduction to Its Methodology , 1980 .

[22]  A. Sheldon The role of parallel function in the acquisition of relative clauses in English , 1973 .

[23]  Pierre Zweigenbaum,et al.  Enriching Medical Terminologies: an Approach Based on Aligned Corpora , 2006, MIE.

[24]  Lynette Hirschman,et al.  A Model-Theoretic Coreference Scoring Scheme , 1995, MUC.

[25]  Juhani Järvikivi,et al.  Ambiguous Pronoun Resolution , 2005, Psychological science.

[26]  Nancy Ide American National Corpus (ANC) , 2002 .

[27]  Richard Evans,et al.  Applying Machine Learning Toward an Automatic Classification of It , 2001, Lit. Linguistic Comput..

[28]  Roland Stuckardt Machine-Learning-Based vs. Manually Designed Approaches to Anaphor Resolution: the Best of Two Worlds , 2002 .

[29]  Shalom Lappin,et al.  An Algorithm for Pronominal Anaphora Resolution , 1994, CL.

[30]  Shirley Dex,et al.  JR 旅客販売総合システム(マルス)における運用及び管理について , 1991 .

[31]  Eugene Charniak,et al.  Toward a model of children's story comprehension , 1972 .

[32]  Ruslan Mitkov,et al.  Evaluation Tool for Rule-based Anaphora Resolution Methods , 2001, ACL.

[33]  Richard Evans A Comparison of Rule-Based and Machine Learning Methods for Identifying Non-nominal It , 2000, Natural Language Processing.

[34]  J. Aiman,et al.  Level of reading difficulty in The American College of Obstetricians and Gynecologsts Patient Education pamphlets , 1989, Obstetrics and gynecology.

[35]  M. C. Templin Certain language skills in children , 1957 .

[36]  Ron Artstein,et al.  The Reliability of Anaphoric Annotation, Reconsidered: Taking Ambiguity into Account , 2005, FCA@ACL.

[37]  Mark A. Przybocki,et al.  The Automatic Content Extraction (ACE) Program – Tasks, Data, and Evaluation , 2004, LREC.

[38]  Hwee Tou Ng,et al.  A Machine Learning Approach to Coreference Resolution of Noun Phrases , 2001, CL.

[39]  C. M. Sperberg-McQueen,et al.  Guidelines for electronic text encoding and interchange , 1994 .

[40]  S. Rankin,et al.  Patient education : principles & practices , 2001 .

[41]  O. Kirksey,et al.  Assessment of Selected Patient Educational Materials of Various Chain Pharmacies , 2004, Journal of health communication.

[42]  Lynette Hirschman,et al.  Automating Coreference: The Role of Annotated Training Data , 1998, ArXiv.

[43]  Yorick Wilks,et al.  University of Sheffield: description of the LaSIE system as used for MUC-6 , 1995, MUC.

[44]  Margaret Comerford Freda,et al.  The readability of American Academy of Pediatrics patient education brochures. , 2005, Journal of pediatric health care : official publication of National Association of Pediatric Nurse Associates & Practitioners.

[45]  Wlodzislaw Duch,et al.  Development of a Pediatric Text-Corpus for Part-of-Speech Tagging , 2004, Intelligent Information Systems.

[46]  Douglas E. Appelt,et al.  SRI International FASTUS SystemMUC-6 Test Results and Analysis , 1995, MUC.

[47]  Breck Baldwin,et al.  CogNIAC: high precision coreference with limited knowledge and linguistic resources , 1997 .

[48]  B. Redman The Practice Of Patient Education , 1996 .

[49]  Scott W. Bennett,et al.  Discourse Tagging Tool and Discourse-Tagged Multilingual Corpora , 1994 .

[50]  Kees van Deemter,et al.  Coreference Annotation: Whither? , 2000, LREC.

[51]  Rebecca J. Passonneau Computing Reliability for Coreference Annotation , 2004, LREC.

[52]  Jacob Cohen A Coefficient of Agreement for Nominal Scales , 1960 .

[53]  Mark W. Davis,et al.  Text Alignment in the Real World: Improving Alignments of Noisy Translations Using Common Lexical Features, String Matching Strategies and N-Gram Comparisons , 1995, EACL.

[54]  S. Griffis EDITOR , 1997, Journal of Navigation.

[55]  D. Biber Methodological Issues Regarding Corpus-based Analyses of Linguistic Variation , 1990 .

[56]  James H. Martin,et al.  Speech and language processing: an introduction to natural language processing , 2000 .

[57]  R J Stevenson,et al.  The use of heuristic strategies in the interpretation of pronouns , 1990, Journal of psycholinguistic research.

[58]  Rudolf Franz Flesch How to Write Plain English , 1981 .

[59]  Krishna Bharat,et al.  SPHINX: A Framework for Creating Personal, Site-Specific Web Crawlers , 1998, Comput. Networks.

[60]  Andrew McCallum,et al.  Toward Conditional Models of Identity Uncertainty with Application to Proper Noun Coreference , 2003, IIWeb.

[61]  Richard Evans,et al.  Enhancing Preference-Based Anaphora Resolution with Genetic Algorithms , 2000, Natural Language Processing.

[62]  J. Hintikka On denoting what? , 2005, Synthese.

[63]  Megumi Kameyama,et al.  Recognizing referential links: an information extraction prespective , 1997, ArXiv.

[64]  Renata Vieira,et al.  Processing definite descriptions in corpora , 2000 .

[65]  Ruslan Mitkov,et al.  Towards a more consistent and comprehensive evaluation of anaphora resolution algorithms and systems , 2001, Appl. Artif. Intell..

[66]  Wjm Levelt PSYCHOLOGY OF LANGUAGE - GERMAN - HORMANN,H , 1968 .

[67]  Daniela B Friedman,et al.  Readability of cancer information on the internet. , 2004, Journal of cancer education : the official journal of the American Association for Cancer Education.

[68]  Candace L. Sidner,et al.  Attention, Intentions, and the Structure of Discourse , 1986, CL.

[69]  Tony McEnery,et al.  Domains, text types, aspect marking and English-Chinese translation , 1999 .

[70]  C. D. Paice,et al.  Towards the automatic recognition of anaphoric features in English text: the impersonal pronoun “it” , 1987 .

[71]  Ruslan Mitkov,et al.  1 Anaphora Resolution : Where Do We Stand Now ? , 2000 .

[72]  R. Mitkov,et al.  Coreference and anaphora: developing annotating tools, annotated resources and annotation strategies , 2000 .

[73]  Kees van Deemter,et al.  On Coreferring: Coreference in MUC and Related Annotation Schemes , 2000, CL.

[74]  Christian Köhler,et al.  Health-related searches on the Internet. , 2004, JAMA.

[75]  David W. Carroll,et al.  Psychology of Language , 1993 .

[76]  Michael Strube,et al.  Annotating Anaphoric and Bridging Relations with MMAX , 2001, SIGDIAL Workshop.

[77]  Christian Chiarcos,et al.  PoCoS - Potsdam Coreference Scheme , 2007, LAW@ACL.

[78]  Ron Artstein,et al.  = Alpha (or Beta) , 2005 .

[79]  Caroline Gasperin,et al.  How useful are similarity word lists for indirect anaphora resolution , 2005 .

[80]  S. Garrod,et al.  The Role of Different Types of Anaphor in the On-Line Resolution of Sentences in a Discourse , 1994 .

[81]  David McKelvie,et al.  The MATE workbench - An annotation tool for XML coded speech corpora , 2001, Speech Commun..

[82]  Mitchell P. Marcus,et al.  Text Chunking using Transformation-Based Learning , 1995, VLC@ACL.

[83]  Donald D. Cowan,et al.  A Physician's Authoring Tool for Generation of Personalized Health Education in Reconstructive Surgery , 2006, AAAI Spring Symposium: Argumentation for Consumers of Healthcare.

[84]  John R. Frederiksen,et al.  Understanding Anaphora: Rules Used by Readers in Assigning Pronominal Referents. Technical Report No. 3, December 1, 1976 through July 1, 1980. , 1981 .

[85]  J. Skelton,et al.  "I" and "we": a concordancing analysis of how doctors and patients use first person pronouns in primary care consultations. , 2002, Family practice.

[86]  Richard Evans,et al.  A New, Fully Automatic Version of Mitkov's Knowledge-Poor Pronoun Resolution Method , 2002, CICLing.

[87]  F. Hobbs,et al.  Concordancing: use of language-based research in medical communication , 1999, The Lancet.

[88]  Michael Strube,et al.  The Influence of Minimum Edit Distance on Reference Resolution , 2002, EMNLP.

[89]  Scott Bennett,et al.  Evaluating Automated and Manual Acquisition of Anaphora Resolution Strategies , 1995, ACL.

[90]  Joel Tetreault,et al.  A Corpus-Based Evaluation of Centering and Pronoun Resolution , 2001, Computational Linguistics.

[91]  Pierre Zweigenbaum,et al.  Using Word Alignment to Extend Multilingual Medical Terminologies , 2006 .

[92]  Mike Scott Wordsmith Tools version 3 , 1997 .

[93]  Andrei Popescu-Belis,et al.  Online Evaluation of Coreference Resolution , 2004, LREC.

[94]  Simon Garrod,et al.  Interpreting anaphoric relations: The integration of semantic information while reading. , 1977 .

[95]  R. Flesch The Art of Readable Writing , 1974 .

[96]  Donna K. Byron,et al.  The Uncommon Denominator: A Proposal for Consistent Reporting of Pronoun Resolution Results , 2001, Computational Linguistics.

[97]  L. Wallace,et al.  American Academy of Family Physicians patient education materials: can patients read them? , 2004, Family medicine.

[98]  Douglas Biber,et al.  Variation across speech and writing: Methodology , 1988 .

[99]  Christiane Fellbaum,et al.  Book Reviews: WordNet: An Electronic Lexical Database , 1999, CL.

[100]  Jeanette K. Gundel,et al.  Cognitive Status and the form of Referring Expressions in Discourse , 1993, The Oxford Handbook of Reference.

[101]  G. Harry McLaughlin,et al.  SMOG Grading - A New Readability Formula. , 1969 .

[102]  Ruslan Mitkov,et al.  COMPARING PRONOUN RESOLUTION ALGORITHMS , 2007, Comput. Intell..

[103]  Lynette Hirschman,et al.  Mixed-Initiative Development of Language Processing Systems , 1997, ANLP.

[104]  Constantin Orasan,et al.  The importance of annotated corpora for NLP: the cases of anaphora resolution and clause splitting , 2000 .

[105]  Rebecca J. Passonneau Applying Reliability Metrics to Co-Reference Annotation , 1997, ArXiv.

[106]  Ellen Riloff,et al.  Corpus-Based Identification of Non-Anaphoric Noun Phrases , 1999, ACL.

[107]  Barbara Di Eugenio,et al.  Squibs and Discussions: The Kappa Statistic: A Second Look , 2004, CL.

[108]  Graeme Hirst,et al.  Anaphora in Natural Language Understanding: A Survey , 1981, Lecture Notes in Computer Science.

[109]  Rafael Muñoz,et al.  An Algorithm for Anaphora Resolution in Spanish Texts , 2001, CL.

[110]  James H. Martin,et al.  Speech and Language Processing: An Introduction to Natural Language Processing, Computational Linguistics, and Speech Recognition , 2000 .

[111]  Paul P. Lee,et al.  Readability of Ocular Medication Inserts , 2003, Journal of glaucoma.

[112]  Beata Beigman Klebanov Using Latent Semantic Analysis for Pronominal Anaphora Resolution , 2001 .

[113]  George Kingsley Zipf,et al.  Human behavior and the principle of least effort , 1949 .

[114]  Donald D. Cowan,et al.  The Development of a Natural Language Generation System for Personalized e-Health Information , 2007 .

[115]  Massimo Poesio,et al.  The MATE meta-scheme for coreference in dialogues in multiple languages , 1999 .

[116]  Branimir Boguraev,et al.  Anaphora for Everyone: Pronominal Anaphora Resolution without a Parser , 1996, COLING.

[117]  Ted Briscoe,et al.  Can Anaphoric Definite Descriptions be Replaced by Pronouns? , 2004, LREC.

[118]  Beth M. Sundheim,et al.  Overview of Results of the MUC-6 Evaluation , 1995, MUC.

[119]  R. Carter,et al.  Applying corpus linguistics in a health care context , 2007 .

[120]  Erhard W. Hinrichs,et al.  A Unified Representation for Morphological, Syntactic, Semantic, and Referential Annotations , 2005, FCA@ACL.

[121]  Ruslan Mitkov,et al.  Robust Pronoun Resolution with Limited Knowledge , 1998, ACL.

[122]  Beatrice Santorini,et al.  Building a Large Annotated Corpus of English: The Penn Treebank , 1993, CL.

[123]  Jaime G. Carbonell,et al.  Anaphora Resolution: A Multi-Strategy Approach , 1988, COLING.

[124]  Jerry R. Hobbs Resolving pronoun references , 1986 .

[125]  Branimir Boguraev,et al.  Introduction to the Special Issue on Computational Anaphora Resolution , 2001, CL.

[126]  G. Murphy Processes of understanding anaphora , 1985 .

[127]  Jean Carletta,et al.  Assessing Agreement on Classification Tasks: The Kappa Statistic , 1996, CL.

[128]  Constantin Orasan,et al.  PALinkA: A highly customisable tool for discourse annotation , 2003, SIGDIAL Workshop.

[129]  Constantin Orasan,et al.  NPs for Events: Experiments in Coreference Annotation , 2006, LREC.

[130]  Constantin Orasan,et al.  Anaphora Resolution Exercise: an Overview , 2008, LREC.