SemEval-2017 Task 6: #HashtagWars: Learning a Sense of Humor

This paper describes a new shared task for humor understanding that attempts to eschew the ubiquitous binary approach to humor detection and focus on comparative humor ranking instead. The task is based on a new dataset of funny tweets posted in response to shared hashtags, collected from the ‘Hashtag Wars’ segment of the TV show @midnight. The results are evaluated in two subtasks that require the participants to generate either the correct pairwise comparisons of tweets (subtask A), or the correct ranking of the tweets (subtask B) in terms of how funny they are. 7 teams participated in subtask A, and 5 teams participated in subtask B. The best accuracy in subtask A was 0.675. The best (lowest) rank edit distance for subtask B was 0.872.

[1]  Renxian Zhang,et al.  Recognizing Humor on Twitter , 2014, CIKM.

[2]  Bianca Zadrozny,et al.  Learning and evaluating classifiers under sample selection bias , 2004, ICML.

[3]  J. Heckman.  Sample selection bias as a specification error , 1979 .

[4]  Diane J. Litman,et al.  Humor: Prosody Analysis and Automatic Recognition for F*R*I*E*N*D*S* , 2006, EMNLP.

[5]  Nikos Pelekis,et al.  DataStories at SemEval-2017 Task 6: Siamese LSTM with Attention for Humorous Text Comparison , 2017, SemEval@ACL.

[6]  Yishay Raz,et al.  Automatic Humor Classification on Twitter , 2012, NAACL.

[7]  Xiaojuan Ma,et al.  SRHR at SemEval-2017 Task 6: Word Associations for Humour Recognition , 2017, SemEval@ACL.

[8]  Paolo Rosso,et al.  A multidimensional approach for detecting irony in Twitter , 2013, Lang. Resour. Evaluation.

[9]  Ted Pedersen,et al.  Duluth at SemEval-2017 Task 6: Language Models in Humor Detection , 2017, SemEval@ACL.

[10]  Diyi Yang,et al.  Humor Recognition and Humor Anchor Extraction , 2015, EMNLP.

[11]  Dragomir R. Radev,et al.  Humor in Collective Discourse: Unsupervised Funniness Detection in the New Yorker Cartoon Caption Contest , 2015, LREC.

[12]  Jan Snajder,et al.  TakeLab at SemEval-2017 Task 6: #RankingHumorIn4Pages , 2017, SemEval@ACL.

[13]  Carlo Strapparava,et al.  Making Computers Laugh: Investigations in Automatic Humor Recognition , 2005, HLT.

[14]  Yuriy Brun,et al.  That's What She Said: Double Entendre Identification , 2011, ACL.

[15]  Xiwu Han,et al.  QUB at SemEval-2017 Task 6: Cascaded Imbalanced Classification for Humor Analysis in Twitter , 2017, *SEMEVAL.

[16]  Mukesh Zaveri,et al.  SVNIT $@$ SemEval 2017 Task-6: Learning a Sense of Humor Using Supervised Approach , 2017, SemEval@ACL.

[17]  Horacio Saggion,et al.  Automatic Detection of Irony and Humour in Twitter , 2014, ICCC.

[18]  Anna Rumshisky,et al.  HumorHawk at SemEval-2017 Task 6: Mixing Meaning and Sound for Humor Recognition , 2017, SemEval@ACL.

[19]  Dafna Shahaf,et al.  Inside Jokes: Identifying Humorous Cartoon Captions , 2015, KDD.

[20]  Anna Rumshisky,et al.  #HashtagWars: Learning a Sense of Humor , 2016, ArXiv.