Sentence and Expression Level Annotation of Opinions in User-Generated Discourse

In this paper, we introduce a corpus of consumer reviews from the rateitall and the eopinions websites annotated with opinion-related information. We present a two-level annotation scheme. In the first stage, the reviews are analyzed at the sentence level for (i) relevancy to a given topic, and (ii) expressing an evaluation about the topic. In the second stage, on-topic sentences containing evaluations about the topic are further investigated at the expression level for pinpointing the properties (semantic orientation, intensity), and the functional components of the evaluations (opinion terms, targets and holders). We discuss the annotation scheme, the inter-annotator agreement for different subtasks and our observations.

[1]  Jacob Cohen.  A Coefficient of Agreement for Nominal Scales , 1960 .

[2]  Klaus Krippendorff,et al.  Content Analysis: An Introduction to Its Methodology , 1980 .

[3]  Jan Svartvik,et al.  A __ comprehensive grammar of the English language , 1988 .

[4]  Ellen Riloff,et al.  Learning Extraction Patterns for Subjective Expressions , 2003, EMNLP.

[5]  Rebecca J. Passonneau.  Computing Reliability for Coreference Annotation , 2004, LREC.

[6]  Bing Liu,et al.  Mining and summarizing customer reviews , 2004, KDD.

[7]  Claire Cardie,et al.  Annotating Expressions of Opinions and Emotions in Language , 2005, Lang. Resour. Evaluation.

[8]  Claire Cardie,et al.  Identifying Sources of Opinions with Conditional Random Fields and Extraction Patterns , 2005, HLT.

[9]  Janyce Wiebe,et al.  Annotating Attributions and Private States , 2005, FCA@ACL.

[10]  Xiaoyan Zhu,et al.  Movie review mining and summarization , 2006, CIKM '06.

[11]  Eduard Hovy,et al.  Extracting Opinions, Opinion Holders, and Topics Expressed in Online News Media Text , 2006 .

[12]  Andrea Esuli,et al.  SENTIWORDNET: A Publicly Available Lexical Resource for Opinion Mining , 2006, LREC.

[13]  Claire Cardie,et al.  Identifying Expressions of Opinion in Context , 2007, IJCAI.

[14]  Feiyu Xu,et al.  Fine-grained Opinion Topic and Polarity Identification , 2008, LREC.

[15]  Philip S. Yu,et al.  A holistic lexicon-based approach to opinion mining , 2008, WSDM '08.

[16]  Nicholas Asher,et al.  Distilling Opinion in Discourse: A Preliminary Study , 2008, COLING.

[17]  Claire Cardie,et al.  Topic Identification for Fine-Grained Opinion Analysis , 2008, COLING.

[18]  Theresa Wilson,et al.  Annotating Subjective Content in Meetings , 2008, LREC.

[19]  Swapna Somasundaran,et al.  Discourse Level Opinion Relations: An Annotation Study , 2008, SIGDIAL Workshop.

[20]  Theresa Wilson.  Fine-grained subjectivity and sentiment analysis: recognizing the intensity, polarity, and attitudes of private states , 2008 .

[21]  Tejashri Inadarchand Jain,et al.  Recognizing Contextual Polarity in Phrase-Level Sentiment Analysis , 2010 .

[22]  Angela Fahrni,et al.  Old Wine or Warm Beer : Target-Specific Sentiment Analysis of Adjectives , .