Annotating Expressions of Opinions and Emotions in Language

This paper describes a corpus annotation project to study issues in the manual annotation of opinions, emotions, sentiments, speculations, evaluations and other private states in language. The resulting corpus annotation scheme is described, as well as examples of its use. In addition, the manual annotation process and the results of an inter-annotator agreement study on a 10,000-sentence corpus of articles drawn from the world press are presented.

[1]  C. Osgood,et al.  The Measurement of Meaning , 1958 .

[2]  J. M. Kittross.  The measurement of meaning , 1959 .

[3]  D. R. Heise,et al.  Semantic di erential profiles for 1000 most frequent English words , 1965 .

[4]  Keith S. Donnellan.  Reference and Definite Descriptions , 1966 .

[5]  W. Quine.  The ways of paradox, and other essays , 1966 .

[6]  Stephen R. Anderson,et al.  A Festschrift for Morris Halle , 1973 .

[7]  J. Rivera.  A structural theory of the emotions , 1977 .

[8]  H. Castañeda.  ON THE PHILOSOPHICAL FOUNDATIONS OF THE THEORY OF COMMUNICATION: REFERENCE1 , 1977 .

[9]  W. Riggan,et al.  Transparent Minds: Narrative Modes for Presenting Consciousness in Fiction , 1978 .

[10]  V. Dijk,et al.  Pragmatics of language and literature , 1978 .

[11]  S. Chatman,et al.  Story and Discourse: Narrative Structure in Fiction and Film , 1979 .

[12]  Janet Dean Fodor,et al.  The linguistic description of opaque contexts , 1979 .

[13]  S. Kuroda.  Where epistemology, style and grammar meet , 1979 .

[14]  S. Kuroda.  Reflections on the foundations of narrative theory, from a linguistic point of view , 1979 .

[15]  Klaus Krippendorff,et al.  Content Analysis: An Introduction to Its Methodology , 1980 .

[16]  J. Russell.  A circumplex model of affect. , 1980 .

[17]  S. Chatman.  Story and Discourse: Narrative Structure in Fiction and Film , 1980 .

[18]  C. W. Anderson,et al.  Computer assisted modeling of affective tone in written documents , 1982, Comput. Humanit..

[19]  Jan Svartvik,et al.  A __ comprehensive grammar of the English language , 1988 .

[20]  Janusz S. Bień,et al.  Beliefs, Points of View, and Multiple Environments , 1983, Cogn. Sci..

[21]  D. Schiffrin.  Meaning, form, and use in context : linguistic applications , 1984 .

[22]  Keith r. Sanders.  Political Communication Yearbook, 1984 , 1985 .

[23]  D. Watson,et al.  Toward a consensual structure of mood. , 1985, Psychological bulletin.

[24]  Michael Halliday,et al.  An Introduction to Functional Grammar , 1985 .

[25]  Gilles Fauconnier,et al.  Mental Spaces: Aspects of Meaning Construction in Natural Language , 1985 .

[26]  William J. Rapaport,et al.  Logical Foundations for Belief Representation , 1986, Cogn. Sci..

[27]  Nicholas Asher,et al.  Belief in discourse representation theory , 1986, J. Philos. Log..

[28]  Andrew Ortony,et al.  The Referential Structure of the Affective Lexicon , 1987, Cogn. Sci..

[29]  P. Johnson-Laird,et al.  The language of emotions: An analysis of a semantic field , 2013 .

[30]  D. Biber,et al.  Styles of stance in English: Lexical and grammatical marking of evidentiality and affect , 1989 .

[31]  J. Wiebe,et al.  References in Narrative Text , 1991 .

[32]  Sabine Bergler,et al.  Evidential analysis of reported speech , 1992 .

[33]  Jeannett Martin,et al.  English Text: System and structure , 1992 .

[34]  Janyce Wiebe,et al.  Tracking Point of View in Narrative , 1994, Comput. Linguistics.

[35]  Monica Amor.  Felix Gonzalez‐Torres 1957–1996 , 1995 .

[36]  S. Onega,et al.  Narratology: An Introduction , 1996 .

[37]  Ellen Spertus,et al.  Smokey: Automatic Recognition of Hostile Messages , 1997, AAAI/IAAI.

[38]  Vasileios Hatzivassiloglou,et al.  Predicting the Semantic Orientation of Adjectives , 1997, ACL.

[39]  Kathleen R. McKeown,et al.  Predicting the semantic orientation of adjectives , 1997 .

[40]  Janyce Wiebe,et al.  Recognizing subjectivity: a case study in manual tagging , 1999, Natural Language Engineering.

[41]  Janyce Wiebe,et al.  Learning Subjective Adjectives from Corpora , 2000, AAAI/IAAI.

[42]  Jeannett Martin.  Beyond Exchange : Appraisal Systems in English , 2000 .

[43]  S. Hunston,et al.  Evaluation in Text: Authorial Stance and the Construction of Discourse , 2001 .

[44]  Pero Subasic,et al.  Affect analysis of text using fuzzy semantic typing , 2001, IEEE Trans. Fuzzy Syst..

[45]  Janyce Wiebe,et al.  A Corpus Study of Evaluative and Speculative Language , 2001, SIGDIAL Workshop.

[46]  Janyce Wiebe,et al.  Identifying Collocations for Recognizing Opinions , 2001 .

[47]  B Daille,et al.  Collocation: Computational Extraction, Analysis and Exploitation , 2001 .

[48]  Bo Pang,et al.  Thumbs up? Sentiment Classification using Machine Learning Techniques , 2002, EMNLP.

[49]  H. Cunningham,et al.  A framework and graphical development environment for robust NLP tools and applications. , 2002, ACL 2002.

[50]  Peter D. Turney.  Thumbs Up or Thumbs Down? Semantic Orientation Applied to Unsupervised Classification of Reviews , 2002, ACL.

[51]  Janyce Wiebe,et al.  Instructions for annotating opinions in newspaper articles , 2002 .

[52]  Claire Cardie,et al.  Combining Low-Level and Summary Representations of Opinions for Multi-Perspective Question Answering , 2003, New Directions in Question Answering.

[53]  Abe Kazemzadeh,et al.  Recognizing Expressions of Commonsense Psychology in English Text , 2003, ACL.

[54]  Razvan C. Bunescu,et al.  Sentiment analyzer: extracting sentiments about a given topic using natural language processing techniques , 2003, Third IEEE International Conference on Data Mining.

[55]  David M. Pennock,et al.  Mining the peanut gallery: opinion extraction and semantic classification of product reviews , 2003, WWW '03.

[56]  Hong Yu,et al.  Towards Answering Opinion Questions: Separating Facts from Opinions and Identifying the Polarity of Opinion Sentences , 2003, EMNLP.

[57]  Michael L. Littman,et al.  Measuring praise and criticism: Inference of semantic orientation from association , 2003, TOIS.

[58]  Ellen Riloff,et al.  Learning subjective nouns using extraction pattern bootstrapping , 2003, CoNLL.

[59]  Proceedings of the 3rd IEEE International Conference on Data Mining (ICDM 2003), 19-22 December 2003, Melbourne, Florida, USA , 2003, ICDM.

[60]  Brian S. Butler,et al.  The Power of Words: Unveiling the Speaker and Writer's Hidden Craft , 2003 .

[61]  Ellen Riloff,et al.  Learning Extraction Patterns for Subjective Expressions , 2003, EMNLP.

[62]  Janyce Wiebe,et al.  Annotating Opinions in the World Press , 2003, SIGDIAL Workshop.

[63]  Claire Cardie,et al.  Playing the Telephone Game: Determining the Hierarchical Structure of Perspective and Speech Expressions , 2004, COLING.

[64]  Laurence R. Horn,et al.  The handbook of pragmatics , 2004 .

[65]  Janyce Wiebe,et al.  Just How Mad Are You? Finding Strong and Weak Opinion Clauses , 2004, AAAI.

[66]  Janyce Wiebe,et al.  Learning Subjective Language , 2004, CL.