The Laughing Machine: Predicting Humor in Video

Humor is a very important communication tool; yet, it is an open problem for machines to understand humor. In this paper, we build a new multimodal dataset for humor prediction that includes subtitles and video frames, as well as humor labels associated with video’s timestamps. On top of it, we present a model to predict whether a subtitle causes laughter. Our model uses the visual modality through facial expression and character name recognition, together with the verbal modality, to explore how the visual modality helps. In addition, we use an attention mechanism to adjust the weight for each modality to facilitate humor prediction. Interestingly, our experimental results show that the performance boost by combinations of different modalities, and the attention mechanism and the model mostly relies on the verbal modality.

[1]  Paolo Rosso,et al.  UO_UPV: Deep Linguistic Humor Detection in Spanish Social Media , 2018, IberEval@SEPLN.

[2]  Pascale Fung,et al.  Deep Learning of Audio and Language Features for Humor Prediction , 2016, LREC.

[3]  Hod Lipson,et al.  Humor as Circuits in Semantic Networks , 2012, ACL.

[4]  Luca Antiga,et al.  Automatic differentiation in PyTorch , 2017 .

[5]  David Matthews,et al.  Unsupervised joke generation from big data , 2013, ACL.

[6]  Yuriy Brun,et al.  That's What She Said: Double Entendre Identification , 2011, ACL.

[7]  Xiaojun Wan,et al.  Multi-Modal Sarcasm Detection in Twitter with Hierarchical Fusion Model , 2019, ACL.

[8]  Rossano Schifanella,et al.  Detecting Sarcasm in Multimodal Social Platforms , 2016, ACM Multimedia.

[9]  Carlo Strapparava,et al.  Making Computers Laugh: Investigations in Automatic Humor Recognition , 2005, HLT.

[10]  Thomas Wolf,et al.  HuggingFace's Transformers: State-of-the-art Natural Language Processing , 2019, ArXiv.

[11]  Louis-Philippe Morency,et al.  UR-FUNNY: A Multimodal Language Dataset for Understanding Humor , 2019, EMNLP.

[12]  Ming-Wei Chang,et al.  BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding , 2019, NAACL.

[13]  John C. Meyer,et al.  Humor as a Double‐Edged Sword: Four Functions of Humor in Communication , 2000 .

[14]  Pascale Fung,et al.  A Long Short-Term Memory Framework for Predicting Humor in Dialogues , 2016, NAACL.

[15]  Pushpak Bhattacharyya,et al.  Predicting Readers' Sarcasm Understandability by Modeling Gaze Behavior , 2016, AAAI.

[16]  Julia Taylor Rayz,et al.  Computationally Recognizing Wordplay in Jokes , 2004 .

[17]  Lizhen Liu,et al.  Exploiting Syntactic Structures for Humor Recognition , 2018, COLING.

[18]  Xiaojuan Ma,et al.  Recognizing Humour using Word Associations and Humour Anchor Extraction , 2018, COLING.

[19]  Victor Raskin,et al.  Semantic mechanisms of humor , 1984 .

[20]  Shelly Sachdeva,et al.  Detection of Sarcasm in Text Data using Deep Convolutional Neural Networks , 2017, Scalable Comput. Pract. Exp..

[21]  Jun Hong,et al.  Sarcasm Detection on Czech and English Twitter , 2014, COLING.

[22]  Diyi Yang,et al.  Humor Recognition and Humor Anchor Extraction , 2015, EMNLP.

[23]  Sergey Ioffe,et al.  Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift , 2015, ICML.

[24]  K. Binsted.  Using Humour to Make Natural Language Interfaces More Friendly , 1995 .

[25]  Kota Yoshida,et al.  Neural Joking Machine : Humorous image captioning , 2018, ArXiv.

[26]  H. Leuthold,et al.  Testing theories of irony processing using eye-tracking and ERPs. , 2014, Journal of experimental psychology. Learning, memory, and cognition.

[27]  Peter Robinson,et al.  OpenFace: An open source facial behavior analysis toolkit , 2016, 2016 IEEE Winter Conference on Applications of Computer Vision (WACV).

[28]  Sanja Fidler,et al.  MovieQA: Understanding Stories in Movies through Question-Answering , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[29]  P. Ekman,et al.  What the face reveals : basic and applied studies of spontaneous expression using the facial action coding system (FACS) , 2005 .

[30]  Verónica Pérez-Rosas,et al.  Towards Multimodal Sarcasm Detection (An _Obviously_ Perfect Paper) , 2019, ACL.

[31]  Tomoaki Ohtsuki,et al.  Sarcasm Detection in Twitter: "All Your Products Are Incredibly Amazing!!!" - Are They Really? , 2014, 2015 IEEE Global Communications Conference (GLOBECOM).

[32]  Diane J. Litman,et al.  Humor: Prosody Analysis and Automatic Recognition for F*R*I*E*N*D*S* , 2006, EMNLP.

[33]  Chong Min Lee,et al.  Predicting Audience's Laughter Using Convolutional Neural Network , 2017 .

[34]  Ellen Riloff,et al.  Sarcasm as Contrast between a Positive Sentiment and Negative Situation , 2013, EMNLP.

[35]  Adriana Kovashka,et al.  Story Understanding in Video Advertisements , 2018, BMVC.

[36]  Jürgen Schmidhuber,et al.  Long Short-Term Memory , 1997, Neural Computation.

[37]  Jeffrey Dean,et al.  Distributed Representations of Words and Phrases and their Compositionality , 2013, NIPS.

[38]  Carlo Strapparava,et al.  HAHAcronym: A Computational Humor System , 2005, ACL.

[39]  Ari Rappoport,et al.  Semi-Supervised Recognition of Sarcasm in Twitter and Amazon , 2010, CoNLL.

[40]  Frank Hutter,et al.  Decoupled Weight Decay Regularization , 2017, ICLR.

[41]  Ruli Manurung,et al.  A practical application of computational humour , 2007 .

[42]  Kim Binsted,et al.  An Implemented Model of Punning Riddles , 1994, AAAI.

[43]  Kenji Araki,et al.  A Complete and Modestly Funny System for Generating and Performing Japanese Stand-Up Comedy , 2008, COLING.