Computational Generation of Referring Expressions: A Survey

This article offers a survey of computational research on referring expression generation (REG). It introduces the REG problem and describes early work in this area, discussing what basic assumptions lie behind it, and showing how its remit has widened in recent years. We discuss computational frameworks underlying REG, and demonstrate a recent trend that seeks to link REG algorithms with well-established Knowledge Representation techniques. Considerable attention is given to recent efforts at evaluating REG algorithms and the lessons that they allow us to learn. The article concludes with a discussion of the way forward in REG, focusing on references in larger and more realistic settings.

[1]  Pascal Van Hentenryck Constraint satisfaction in logic programming , 1989, Logic programming.

[2]  A. Kilgarriff,et al.  Thesauruses for natural language processing , 2003, International Conference on Natural Language Processing and Knowledge Engineering, 2003. Proceedings. 2003.

[3]  Simone Paolo Ponzetto,et al.  Collaboratively built semi-structured content and Artificial Intelligence: The story so far , 2013, Artif. Intell..

[4]  Roser Morante,et al.  CNTS: Memory-Based Learning of Generating Repeated References , 2008, INLG.

[5]  Emiel Krahmer,et al.  What Computational Linguists Can Learn from Psychologists (and Vice Versa) , 2010, Computational Linguistics.

[6]  Scott Weinstein,et al.  Centering: A Framework for Modeling the Local Coherence of Discourse , 1995, CL.

[7]  Henry Lieberman,et al.  Beating Common Sense into Interactive Applications , 2004, AI Mag..

[8]  M. Pickering,et al.  Linguistic alignment between people and computers , 2010 .

[9]  Kathleen F. McCoy,et al.  Generating Anaphoric Expressions: Pronoun or Definite Description? , 1999 .

[10]  Marilyn A. Walker,et al.  Individual and Domain Adaptation in Sentence Planning for Dialogue , 2007, J. Artif. Intell. Res..

[11]  Jeanette K. Gundel,et al.  Cognitive Status and the Form of Referring Expressions in Discourse , 1993 .

[12]  Claire Gardent,et al.  GENERATING BRIDGING DEFINITE DESCRIPTIONS , 2008 .

[13]  W. Kintsch,et al.  Language and comprehension , 1982 .

[14]  A. Kilgarriff,et al.  Disambiguating coordinations using word distribution information , 2005 .

[15]  Peter Norvig,et al.  Artificial Intelligence: A Modern Approach , 1995 .

[16]  P. Piwek,et al.  'Proximal' and 'distal' in language and cognition : Evidence from deictic demonstratives in Dutch , 2008 .

[17]  Rebecca J. Passonneau,et al.  Measuring Agreement on Set-valued Items (MASI) for Semantic and Pragmatic Annotation , 2006, LREC.

[18]  S. Brennan,et al.  When conceptual pacts are broken: Partner-specific effects on the comprehension of referring expressions , 2003 .

[19]  Diego Calvanese,et al.  The Description Logic Handbook: Theory, Implementation, and Applications , 2003, Description Logic Handbook.

[20]  Ani Nenkova,et al.  References to Named Entities: a Corpus Study , 2003, HLT-NAACL.

[21]  Iris Hendrickx,et al.  GRAPH: The Costs of Redundancy in Referring Expressions , 2008, INLG.

[22]  Jean-François Baget,et al.  Extensions of Simple Conceptual Graphs: the Complexity of Rules and Constraints , 2011, J. Artif. Intell. Res..

[23]  John F. Sowa,et al.  Conceptual Structures: Information Processing in Mind and Machine , 1983 .

[24]  A. Koller,et al.  Speech Acts: An Essay in the Philosophy of Language , 1969 .

[25]  Albert Gatt,et al.  Generating coherent references to multiple entities , 2007 .

[26]  Laura Stoia,et al.  Noun Phrase Generation for Situated Dialogs , 2006, INLG.

[27]  Frederick A. Olafson,et al.  THE CONCEPT OF BEING , 1987 .

[28]  Robert Dale,et al.  Evaluation in Natural Language Generation: Lessons from Referring Expression Generation , 2007, TAL.

[29]  Karl G. D. Bailey,et al.  Do speakers and listeners observe the Gricean Maxim of Quantity , 2006 .

[30]  Alex Pentland,et al.  Learning words from sights and sounds: a computational model , 2002, Cogn. Sci..

[31]  Johanna D. Moore,et al.  Generating Tailored, Comparative Descriptions with Contextually Appropriate Intonation , 2010, CL.

[32]  Robert Dale,et al.  Generating Referring Expressions Involving Relations , 1991, EACL.

[33]  Johanna D. Moore,et al.  The agreement process: an empirical investigation of human-human computer-mediated collaborative dialogs , 2000, Int. J. Hum. Comput. Stud..

[34]  Moreno I. Coco,et al.  The Impact of Visual Information on Reference Assignment in Sentence Production , 2009 .

[35]  Belle Liang,et al.  Warts and All , 2004 .

[36]  G. Clark,et al.  Reference , 2008 .

[37]  M. Pickering,et al.  Toward a mechanistic psychology of dialogue , 2004, Behavioral and Brain Sciences.

[38]  Robert Dale,et al.  The Use of Spatial Relations in Referring Expression Generation , 2008, INLG.

[39]  D. Roy Grounding words in perception and action: computational insights , 2005, Trends in Cognitive Sciences.

[40]  Emiel Krahmer,et al.  Context sensitive generation of descriptions , 1998, ICSLP.

[41]  Douglas E. Appelt,et al.  Planning English Sentences , 1988, Cogn. Sci..

[42]  David S Wooding,et al.  Eye movements of large populations: I. Implementation and performance of an autonomous public eye tracker , 2002, Behavior research methods, instruments, & computers : a journal of the Psychonomic Society, Inc.

[43]  Kees van Deemter,et al.  Reference and the facilitation of search in spatial domains , 2014 .

[44]  Michael J. Spivey,et al.  Linguistically Mediated Visual Search , 2001, Psychological science.

[45]  Ted Pedersen,et al.  WordNet::Similarity - Measuring the Relatedness of Concepts , 2004, NAACL.

[46]  Ehud Reiter,et al.  Using Spatial Reference Frames to Generate Grounded Textual Summaries of Georeferenced Data , 2008, INLG.

[47]  Philip Resnik,et al.  Using Information Content to Evaluate Semantic Similarity in a Taxonomy , 1995, IJCAI.

[48]  M. Swerts,et al.  The Effects of Visual Beats on Prosodic Prominence: Acoustic Analyses, Auditory Perception and Visual Perception. , 2007 .

[49]  Ronald J. Brachman,et al.  An Overview of the KL-ONE Knowledge Representation System , 1985, Cogn. Sci..

[50]  R. Quirk A Grammar of contemporary English , 1974 .

[51]  Johanna D. Moore,et al.  The First Challenge on Generating Instructions in Virtual Environments , 2010, Empirical Methods in Natural Language Generation.

[52]  Michael Jackson Black Or White , 2011 .

[53]  Stefan Kopp,et al.  An Alignment-Capable Microplanner for Natural Language Generation , 2009, ENLG.

[54]  Helmut Horacek Generating Referential Descriptions Under Conditions of Uncertainty , 2005, ENLG.

[55]  Rob Malouf,et al.  The Order of Prenominal Adjectives in Natural Language Generation , 2000, ACL.

[56]  Helmut Horacek,et al.  On Referring to Sets of Objects Naturally , 2004, INLG.

[57]  Albert Gatt,et al.  The GREC Challenge 2008: Overview and Evaluation Results , 2008, INLG.

[58]  Mija M. Van Der Wege Lexical Entrainment and Lexical Differentiation in Reference Phrase Choice. , 2009 .

[59]  C. Koch,et al.  A saliency-based search mechanism for overt and covert shifts of visual attention , 2000, Vision Research.

[60]  Graeme Hirst,et al.  Collaborating on Referring Expressions , 1991, CL.

[61]  Robert Dale,et al.  Cooking Up Referring Expressions , 1989, ACL.

[62]  Margaret Mitchell,et al.  Class-Based Ordering of Prenominal Modifiers , 2009, ENLG.

[63]  H. H. Clark,et al.  Changing Ideas about Reference , 2004 .

[64]  Amanda Stent,et al.  Automatic Evaluation of Referring Expression Generation Using Corpora ∗ , 2005 .

[65]  Robert Dale,et al.  Algorithms for Generating Referring Expressions: Do They Do What People Do? , 2006, INLG.

[66]  John D. Kelleher,et al.  Incremental Generation of Spatial Referring Expressions in Situated Dialog , 2006, ACL.

[67]  G. Kerdiles,et al.  Saying It with Pictures: a logical landscape of conceptual graphs , 2001 .

[68]  David Cohen,et al.  Being a Man , 1990 .

[69]  Kees van Deemter,et al.  Generation of Referring Expressions: Managing Structural Ambiguities , 2008, COLING.

[70]  James C. Lester,et al.  Deictic Believability: Coordinated Gesture, Locomotion, and Speech in Lifelike Pedagogical Agents , 1999, Appl. Artif. Intell..

[71]  Deb Roy,et al.  Grounded Semantic Composition for Visual Scenes , 2011, J. Artif. Intell. Res..

[72]  K. Solomon,et al.  The Older Man , 1982 .

[73]  Eva Hajičová,et al.  Issues of Sentence Structure and Discourse Patterns. , 1993 .

[74]  Vipin Kumar,et al.  Algorithms for Constraint-Satisfaction Problems: A Survey , 1992, AI Mag..

[75]  Amy Isard,et al.  Situated Reference in a Hybrid Human-Robot Interaction System , 2010, INLG.

[76]  Richard Power,et al.  Optimizing Referential Coherence in Text Generation , 2004, CL.

[77]  V. Ferreira,et al.  Don't Talk About Pink Elephants! , 2006, Psychological science.

[78]  Siobhan Chapman Logic and Conversation , 2005 .

[79]  Robert Dale,et al.  Computational Interpretations of the Gricean Maxims in the Generation of Referring Expressions , 1995, Cogn. Sci..

[80]  Salim Roukos,et al.  Bleu: a Method for Automatic Evaluation of Machine Translation , 2002, ACL.

[81]  Vladimir I. Levenshtein,et al.  Binary codes capable of correcting deletions, insertions, and reversals , 1965 .

[82]  Karen Spärck Jones,et al.  Readings in natural language processing , 1986 .

[83]  Emiel Krahmer,et al.  Introducing shared task evaluation to NLG : The TUNA shared task evaluation challenges , 2010 .

[84]  Jeanette K. Gundel,et al.  Cognitive Status and the form of Referring Expressions in Discourse , 1993, The Oxford Handbook of Reference.

[85]  Alexandre Denis Generating Referring Expressions with Reference Domain Theory , 2010, INLG.

[86]  Ehud Reiter,et al.  The Computational Complexity of Avoiding Conversational Implicatures , 1990, ACL.

[87]  Jeff Z. Pan,et al.  Charting the Potential of Description Logic for the Generation of Referring Expressions , 2010, INLG.

[88]  Martha E. Pollack,et al.  Overloading Intentions for Efficient Practical Reasoning , 1991 .

[89]  Dekang Lin,et al.  An Information-Theoretic Definition of Similarity , 1998, ICML.

[90]  Oliver Lemon,et al.  Learning Lexical Alignment Policies for Generating Referring Expressions for Spoken Dialogue Systems , 2009, ENLG.

[91]  Emiel Krahmer,et al.  Attribute preference and priming in reference production: Experimental evidence and computational modeling , 2011, CogSci.

[92]  Albert Gatt,et al.  Generating Referring Expressions in Context: The GREC Task Evaluation Challenges , 2010, Empirical Methods in Natural Language Generation.

[93]  H. H. Clark,et al.  Audience Design in Meaning and Reference , 1982 .

[94]  Jon Oberlander Do the Right Thing ... but Expect the Unexpected , 1998, Comput. Linguistics.

[95]  Ielka van der Sluis,et al.  Building a Semantically Transparent Corpus for the Generation of Referring Expressions. , 2006, INLG.

[96]  Kees van Deemter Generating Referring Expressions: Boolean Extensions of the Incremental Algorithm , 2002, CL.

[97]  James Shaw,et al.  Ordering Among Premodifiers , 1999, ACL.

[98]  Douglas B. Lenat,et al.  CYC: a large-scale investment in knowledge infrastructure , 1995, CACM.

[99]  Eduard H. Hovy,et al.  Automatic Evaluation of Summaries Using N-gram Co-occurrence Statistics , 2003, NAACL.

[100]  Benny Davis,et al.  That's Nice , 1919 .

[101]  Alexander Koller,et al.  Combining symbolic and corpus-based approaches for the generation of successful referring expressions , 2011, ENLG.

[102]  Sharon C. Salveter Review of Conceptual structures: information processing in mind and machine by John F. Sowa. Addison-Wesley 1984. , 1986 .

[103]  Albert Gatt,et al.  Automatic generation of textual summaries from neonatal intensive care data , 2009 .

[104]  Eleanor Rosch,et al.  Principles of Categorization , 1978 .

[105]  Hector J. Levesque,et al.  Speech Acts and Rationality , 1985, ACL.

[106]  Remko Scha,et al.  Multi-Level Plurals and Distributivity , 1988, ACL.

[107]  Judith Masthoff,et al.  Generating Referring Expressions: Making Referents Easy to Identify , 2007, Computational Linguistics.

[108]  Ehud Reiter,et al.  Generating Approximate Geographic Descriptions , 2009, ENLG.

[109]  Robert Dale,et al.  A Fast Algorithm for the Generation of Referring Expressions , 1992, COLING.

[110]  Claire Gardent,et al.  Generating Minimal Definite Descriptions , 2002, ACL.

[111]  Robert Dale,et al.  Viewing Referring Expression Generation as Search , 2005, IJCAI.

[112]  Albert Gatt,et al.  The TUNA Challenge 2008: Overview and Evaluation Results , 2008, INLG.

[113]  Douglas E. Appelt,et al.  A Computational Model of Referring , 1987, IJCAI.

[114]  Kees van Deemter Generating Referring Expressions that Involve Gradable Properties , 2006, CL.

[115]  R. Passonneau Using Centering to Relax Gricean Informational Constraints on Discourse Anaphoric Noun Phrases , 1996 .

[116]  Richard I. Kittredge,et al.  Using natural-language processing to produce weather forecasts , 1994, IEEE Expert.

[117]  S. Sonnenschein The effects of redundant communications on listeners: Why different types may have different effects , 1984, Journal of psycholinguistic research.

[118]  Albert Gatt,et al.  Introducing Shared Tasks to NLG: The TUNA Shared Task Evaluation Challenges , 2010, Empirical Methods in Natural Language Generation.

[119]  D. Barr,et al.  Limits on theory of mind use in adults , 2003, Cognition.

[120]  Jan Tore Lønning,et al.  Plurals and Collectivity , 1997, Handbook of Logic and Language.

[121]  Pamela W. Jordan,et al.  Intentional influences on object redescriptions in dialogue: evidence from an empirical study , 2000 .

[122]  Ielka van der Sluis,et al.  Generation of Referring Expressions: Assessing the Incremental Algorithm , 2012, Cogn. Sci..

[123]  Emiel Krahmer,et al.  A New Computational Model of Alignment and Overspecification in Reference , 2010 .

[124]  Anne H. Anderson,et al.  The Hcrc Map Task Corpus , 1991 .

[125]  E Karg,et al.  Old man. , 1985, Journal of gerontological nursing.

[126]  Perry A. Zirkel Don't Touch! , 2000 .

[127]  Emiel Krahmer,et al.  Graphs and Booleans: on the Generation of Referring Expressions , 2008 .

[128]  James C. Lester,et al.  Pronominalization in Generated Discourse and Dialogue , 2002, ACL.

[129]  Ani Nenkova,et al.  Information Status Distinctions and Referring Expressions: An Empirical Study of References to People in News Summaries , 2011, CL.

[130]  Kees van Deemter,et al.  Lexical Choice and Conceptual Perspective in the Generation of Plural Referring Expressions , 2007, J. Log. Lang. Inf..

[131]  P. Maes How to Do the Right Thing , 1989 .

[132]  D R Olson,et al.  Language and thought: aspects of a cognitive theory of semantics. , 1970, Psychological review.

[133]  Sameer Pradhan,et al.  Evaluation Metrics , 2007 .

[134]  Martijn Goudbeek,et al.  The Effect of Scene Variation on the Redundant Use of Color in Definite Reference , 2013, Cogn. Sci..

[135]  Chris Mellish,et al.  A Reference Architecture for Natural Language Generation Systems , 2006, Natural Language Engineering.

[136]  Robert Dale,et al.  Building Natural Language Generation Systems: Figures , 2000 .

[137]  P. Jaccard,et al.  Etude comparative de la distribution florale dans une portion des Alpes et des Jura , 1901 .

[138]  Emiel Krahmer,et al.  Need I Say More? On Factors Causing Referential Overspecification (poster) , 2009 .

[139]  David DeVault,et al.  Natural Language Generation and Discourse Context: Computing Distractor Sets from the Focus Stack , 2004, FLAIRS.

[140]  L. R. Dice Measures of the Amount of Ecologic Association Between Species , 1945 .

[141]  H. H. Clark,et al.  Conceptual pacts and lexical choice in conversation. , 1996, Journal of experimental psychology. Learning, memory, and cognition.

[142]  J. Henderson Human gaze control during real-world scene perception , 2003, Trends in Cognitive Sciences.

[143]  Jennifer E. Arnold,et al.  Reference production: Production-internal and addressee-oriented processes , 2008 .

[144]  Kees van Deemter,et al.  Logical Form Equivalence: the Case of Referring Expressions Generation , 2001, EWNLG@ACL.

[145]  Matthew Stone,et al.  Sentence generation as a planning problem , 2007, ACL.

[146]  Robert Dale Generating referring expressions - constructing descriptions in a domain of objects and processes , 1992, ACL-MIT press series in natural language processing.

[147]  Emiel Krahmer,et al.  Graph-Based Generation of Referring Expressions , 2003, CL.

[148]  Kees van Deemter,et al.  Information sharing : reference and presupposition in language generation and interpretation , 2002 .

[149]  Ehud Reiter,et al.  Book Reviews: Building Natural Language Generation Systems , 2000, CL.

[150]  Alexander Koller,et al.  Referring Expressions as Formulas of Description Logic , 2008, INLG.

[151]  John E. Hopcroft,et al.  An n log n algorithm for minimizing states in a finite automaton , 1971 .

[152]  Stuart J. Russell,et al.  Do the right thing , 1991 .

[153]  Marilyn A. Walker,et al.  Learning Content Selection Rules for Generating Object Descriptions in Dialogue , 2005, J. Artif. Intell. Res..

[154]  Albert Gatt,et al.  Intrinsic vs. Extrinsic Evaluation Measures for Referring Expression Generation , 2008, ACL.

[155]  Matthew Stone,et al.  On identifying sets , 2000, INLG.

[156]  Ielka van der Sluis,et al.  Evaluating Algorithms for the Generation of Referring Expressions: Going beyond Toy Domains. , 2007 .

[157]  W. Chafe The Pear Stories: Cognitive, Cultural and Linguistic Aspects of Narrative Production , 1980 .

[158]  Emiel Krahmer,et al.  Does Size Matter - How Much Data is Required to Train a REG Algorithm? , 2011, ACL.

[159]  Stefan Kopp,et al.  Multimodal Communication from Multimodal Thinking - towards an Integrated Model of Speech and Gesture Production , 2008, Int. J. Semantic Comput..

[160]  Norman I. Badler,et al.  Creating Interactive Virtual Humans: Some Assembly Required , 2002, IEEE Intell. Syst..

[161]  B. Keysar,et al.  When do speakers take into account common ground? , 1996, Cognition.

[162]  Hua Cheng,et al.  Integrating Referring and Informing in NP Planning , 1998 .

[163]  Advaith Siddharthan,et al.  Generating Referring Expressions in Open Domains , 2004, ACL.

[164]  Albert Gatt,et al.  The TUNA-REG Challenge 2009: Overview and Evaluation Results , 2009, ENLG.

[165]  Robert Dale,et al.  Generating Subsequent Reference in Shared Visual Scenes: Computation vs Re-Use , 2011, EMNLP.

[166]  Emiel Krahmer,et al.  Generating Multimodal References , 2007 .

[167]  M. Meteer Bridging the generation gap between text planning and linguistic realization , 1991 .

[168]  Anja Belz That's Nice What Can You Do With It? , 2009, Computational Linguistics.

[169]  Markus Guhe,et al.  Adapting referring expressions to the task environment , 2008 .

[170]  W. Levelt,et al.  Viewing and naming objects: eye movements during noun phrase production , 1998, Cognition.

[171]  Srinivas Bangalore,et al.  Evaluation Metrics for Generation , 2000, INLG.

[172]  S. Brennan,et al.  Speakers' eye gaze disambiguates referring expressions early during face-to-face conversation , 2007 .

[173]  Candace L. Sidner,et al.  Attention, Intentions, and the Structure of Discourse , 1986, CL.

[174]  Madalina Croitoru,et al.  A Conceptual Graph Approach for the Generation of Referring Expressions , 2007, IJCAI.

[175]  Terry Winograd,et al.  Understanding natural language , 1974 .

[176]  David S. Johnson,et al.  Computers and Intractability: A Guide to the Theory of NP-Completeness , 1978 .

[177]  Barbara Di Eugenio,et al.  Centering: A Parametric Theory and Its Instantiations , 2004, Computational Linguistics.

[178]  Emiel Krahmer,et al.  Preferences versus Adaptation during Referring Expression Generation , 2010, ACL.

[179]  Matthew Stone,et al.  Textual Economy Through Close Coupling of Syntax and Semantics , 1998, INLG.

[180]  Emiel Krahmer,et al.  Controlling Redundancy in Referring Expressions , 2008, LREC.

[181]  Matthew Stone,et al.  Microplanning with Communicative Intentions: The SPUD System , 2001, Comput. Intell..

[182]  Douglas E. Appelt,et al.  Planning English Referring Expressions , 1985, Artif. Intell..

[183]  Robert Dale,et al.  Attribute-Centric Referring Expression Generation , 2010, Empirical Methods in Natural Language Generation.

[184]  T. Pechmann Incremental speech production and referential overspecification , 1989 .

[185]  Ielka van der Sluis,et al.  Assessing the Incremental Algorithm: A Response to Krahmer et al , 2012, Cogn. Sci..

[186]  George R. Doddington,et al.  Automatic Evaluation of Machine Translation Quality Using N-gram Co-Occurrence Statistics , 2002 .

[187]  Derek G. Corneil,et al.  The graph isomorphism disease , 1977, J. Graph Theory.

[188]  Edward J. McCluskey,et al.  Introduction to the theory of switching circuits , 1965 .

[189]  Kees van Deemter Not Exactly: In Praise of Vagueness , 2010 .

[190]  Robert Dale,et al.  Dialogue Reference in a Visual Domain , 2010, LREC.

[191]  Helmut Horacek,et al.  An Algorithm for Generating Referential Descriptions with Flexible Interfaces , 1997, ACL.

[192]  Ehud Reitert,et al.  THE COMPUTATIONAL COMPLEXITY OF AVOIDING CONVERSATIONAL IMPLICATURES , 1990, ACL 1990.

[193]  Massimo Poesio,et al.  Pronominalization revisited , 2000, COLING.

[194]  Takenobu Tokunaga,et al.  A task-performance evaluation of referring expressions in situated collaborative task dialogues , 2013, Language Resources and Evaluation.

[195]  Josef van Genabith,et al.  Dynamically structuring, updating and interrelating representations of visual and linguistic discourse context , 2005, Artif. Intell..

[196]  Ielka van der Sluis,et al.  Evaluating algorithms for the Generation of Referring Expressions using a balanced corpus , 2007, ENLG.

[197]  Renata Vieira,et al.  A Corpus-based Investigation of Definite Description Use , 1997, CL.