Annotations for Opinion Mining Evaluation in the Industrial Context of the DOXA project

After presenting opinion and sentiment analysis state of the art and the DOXA project, we review the few evaluation campaigns that have dealt in the past with opinion mining. Then we present the two level opinion and sentiment model that we will use for evaluation in the DOXA project and the annotation interface we use for hand annotating a reference corpus. We then present the corpus which will be used on DOXA and report on the hand-annotation task on a corpus of comments on video games and the solution adopted to obtain a sufficient level of inter-annotator agreement.

[1]  Dan Jurafsky,et al.  Automatic Extraction of Opinion Propositions and their Holders , 2004 .

[2]  Martha Palmer,et al.  Verb Semantics and Lexical Selection , 1994, ACL.

[3]  Claire Cardie,et al.  Annotating Expressions of Opinions and Emotions in Language , 2005, Lang. Resour. Evaluation.

[4]  Jérôme Azé,et al.  Présentation de DEFT 06 (DÉfi Fouille de Textes) , 2006 .

[5]  David M. Pennock,et al.  Mining the peanut gallery: opinion extraction and semantic classification of product reviews , 2003, WWW '03.

[6]  Hong Yu,et al.  Towards Answering Opinion Questions: Separating Facts from Opinions and Identifying the Polarity of Opinion Sentences , 2003, EMNLP.

[7]  Yvette Yannnick Mathieu.  Les verbes de sentiment : de l'analyse linguistique au traitement automatique , 2000 .

[8]  Claire Cardie,et al.  Identifying Sources of Opinions with Conditional Random Fields and Extraction Patterns , 2005, HLT.

[9]  Michael L. Littman,et al.  Measuring praise and criticism: Inference of semantic orientation from association , 2003, TOIS.

[10]  Mireille Brigaudiot.  Sciences du langage , 2012 .

[11]  William Pinnock.  A comprehensive grammar of the English language , 2009 .

[12]  Andrea Esuli,et al.  SENTIWORDNET: A Publicly Available Lexical Resource for Opinion Mining , 2006, LREC.

[13]  Ellen Riloff,et al.  Learning subjective nouns using extraction pattern bootstrapping , 2003, CoNLL.

[14]  Jan Svartvik,et al.  A __ comprehensive grammar of the English language , 1988 .

[15]  Peter D. Turney.  Thumbs Up or Thumbs Down? Semantic Orientation Applied to Unsupervised Classification of Reviews , 2002, ACL.

[16]  Swapna Somasundaran,et al.  Discourse Level Opinion Relations: An Annotation Study , 2008, SIGDIAL Workshop.

[17]  Nigel Collier,et al.  Sentiment Analysis using Support Vector Machines with Diverse Information Sources , 2004, EMNLP.

[18]  Siddharth Patwardhan,et al.  Feature Subsumption for Opinion Analysis , 2006, EMNLP.

[19]  Gérard Dray,et al.  Détection d'opinion : Apprenons les bons adjectifs ! , 2008 .

[20]  M. de Rijke,et al.  UvA-DARE ( Digital Academic Repository ) Using WordNet to measure semantic orientations of adjectives , 2004 .

[21]  Jeannett Martin,et al.  The Language of Evaluation: Appraisal in English , 2005 .

[22]  Patrick Paroubek,et al.  Human Judgement as a Parameter in Evaluation Campaigns , 2008, COLING 2008.

[23]  Philip V. Ogren,et al.  Knowtator: A Protégé plug-in for annotated corpus construction , 2006, NAACL.

[24]  Soo-Min Kim,et al.  Identifying and Analyzing Judgment Opinions , 2006, NAACL.

[25]  Claire Cardie,et al.  Evaluating an Opinion Annotation Scheme Using a New Multi-Perspective Question and Answer Corpus , 2006, Computing Attitude and Affect in Text.