Towards Multimodal Sarcasm Detection (An _Obviously_ Perfect Paper)

Sarcasm is often expressed through several verbal and non-verbal cues, e.g., a change of tone, overemphasis in a word, a drawn-out syllable, or a straight looking face. Most of the recent work in sarcasm detection has been carried out on textual data. In this paper, we argue that incorporating multimodal cues can improve the automatic classification of sarcasm. As a first step towards enabling the development of multimodal approaches for sarcasm detection, we propose a new sarcasm dataset, Multimodal Sarcasm Detection Dataset (MUStARD), compiled from popular TV shows. MUStARD consists of audiovisual utterances annotated with sarcasm labels. Each utterance is accompanied by its context of historical utterances in the dialogue, which provides additional information on the scenario where the utterance occurs. Our initial results show that the use of multimodal information can reduce the relative error rate of sarcasm detection by up to 12.9% in F-score when compared to the use of individual modalities. The full dataset is publicly available for use at this https URL

[1]  D. Voyer,et al.  Context and Intonation in the Perception of Sarcasm , 2011 .

[2]  Jian Sun,et al.  Deep Residual Learning for Image Recognition , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[3]  Ari Rappoport,et al.  Semi-Supervised Recognition of Sarcasm in Twitter and Amazon , 2010, CoNLL.

[4]  Marc D. Pell,et al.  The sound of sarcasm , 2008, Speech Commun..

[5]  H. Leuthold,et al.  Testing theories of irony processing using eye-tracking and ERPs. , 2014, Journal of experimental psychology. Learning, memory, and cognition.

[6]  Jeffrey Pennington,et al.  GloVe: Global Vectors for Word Representation , 2014, EMNLP.

[7]  Antal van den Bosch,et al.  The perfect solution for detecting sarcasm in tweets #not , 2013, WASSA@NAACL-HLT.

[8]  Ellen Riloff,et al.  Sarcasm as Contrast between a Positive Sentiment and Negative Situation , 2013, EMNLP.

[9]  Gaël Varoquaux,et al.  Scikit-learn: Machine Learning in Python , 2011, J. Mach. Learn. Res..

[10]  G. Bryant.  Prosodic Contrasts in Ironic Speech , 2010 .

[11]  Erik Cambria,et al.  Tensor Fusion Network for Multimodal Sentiment Analysis , 2017, EMNLP.

[12]  Eduard Hovy,et al.  Emotion Recognition in Conversation: Research Challenges, Datasets, and Recent Advances , 2019, IEEE Access.

[13]  Yaser Sheikh,et al.  OpenPose: Realtime Multi-Person 2D Pose Estimation Using Part Affinity Fields , 2018, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[14]  Pushpak Bhattacharyya,et al.  How Do Cultural Differences Impact the Quality of Sarcasm Annotation?: A Case Study of Indian Annotators and American Text , 2016, LaTeCH@ACL.

[15]  Rada Mihalcea,et al.  MELD: A Multimodal Multi-Party Dataset for Emotion Recognition in Conversations , 2018, ACL.

[16]  Tony Veale,et al.  Detecting Ironic Intent in Creative Comparisons , 2010, ECAI.

[17]  Erik Cambria,et al.  A Deeper Look into Sarcastic Tweets Using Deep Convolutional Neural Networks , 2016, COLING.

[18]  I. Poggi,et al.  Multimodal markers of irony and sarcasm , 2003 .

[19]  Byron C. Wallace,et al.  Humans Require Context to Infer Ironic Intent (so Computers Probably do, too) , 2014, ACL.

[20]  Christopher Potts,et al.  Representing Social Media Users for Sarcasm Detection , 2018, EMNLP.

[21]  Ming-Wei Chang,et al.  BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding , 2019, NAACL.

[22]  David Bamman,et al.  Capturing, Representing, and Interacting with Laughter , 2018, CHI.

[23]  David Bamman,et al.  Contextualized Sarcasm Detection on Twitter , 2015, ICWSM.

[24]  Dirk Hovy,et al.  Putting Sarcasm Detection into Context: The Effects of Class Imbalance and Manual Labelling on Supervised Machine Classification of Twitter Conversations , 2016, ACL.

[25]  Rada Mihalcea,et al.  CASCADE: Contextual Sarcasm Detection in Online Discussion Forums , 2018, COLING.

[26]  Pushpak Bhattacharyya,et al.  Learning Cognitive Features from Gaze Data for Sentiment and Sarcasm Classification using Convolutional Neural Network , 2017, ACL.

[27]  Rada Mihalcea,et al.  DialogueRNN: An Attentive RNN for Emotion Detection in Conversations , 2018, AAAI.

[28]  Pushpak Bhattacharyya,et al.  Harnessing Sequence Labeling for Sarcasm Detection in Dialogue from TV Series ‘Friends’ , 2016, CoNLL.

[29]  Mário J. Silva,et al.  Clues for detecting irony in user-generated contents: oh...!! it's "so easy" ;-) , 2009, TSA@CIKM.

[30]  H. Hotelling.  Relations Between Two Sets of Variates , 1936 .

[31]  Pushpak Bhattacharyya,et al.  Harnessing Cognitive Features for Sarcasm Detection , 2016, ACL.

[32]  Rossano Schifanella,et al.  Detecting Sarcasm in Multimodal Social Platforms , 2016, ACM Multimedia.

[33]  H. Leuthold,et al.  Emotional responses to irony and emoticons in written language: Evidence from EDA and facial EMG , 2016, Psychophysiology.

[34]  Pushpak Bhattacharyya,et al.  Predicting Readers' Sarcasm Understandability by Modeling Gaze Behavior , 2016, AAAI.

[35]  Reza Zafarani,et al.  Sarcasm Detection on Twitter: A Behavioral Modeling Approach , 2015, WSDM.

[36]  David R. Traum,et al.  "yeah Right": Sarcasm Recognition for Spoken Dialogue Systems , 2006, INTERSPEECH.

[37]  Byron C. Wallace,et al.  Modelling Context with User Embeddings for Sarcasm Detection in Social Media , 2016, CoNLL.

[38]  Jens Sadowski,et al.  Comparison of Support Vector Machine and Artificial Neural Network Systems for Drug/Nondrug Classification , 2003, J. Chem. Inf. Comput. Sci..

[39]  Li Fei-Fei,et al.  ImageNet: A large-scale hierarchical image database , 2009, CVPR.

[40]  P. Rockwell,et al.  Lower, Slower, Louder: Vocal Cues of Sarcasm , 2000 .

[41]  Byron C. Wallace,et al.  Sparse, Contextually Informed Models for Irony Detection: Exploiting User Communities, Entities and Sentiment , 2015, ACL.

[42]  Pushpak Bhattacharyya,et al.  Harnessing Context Incongruity for Sarcasm Detection , 2015, ACL.