The Truth and Nothing But the Truth: Multimodal Analysis for Deception Detection

We propose a data-driven method for automatic deception detection in real-life trial data using visual and verbal cues. Using OpenFace with facial action unit recognition, we analyze the movement of facial features of the witness when posed with questions and the acoustic patterns using OpenSmile. We then perform a lexical analysis on the spoken words, emphasizing the use of pauses and utterance breaks, feeding that to a Support Vector Machine to test deceit or truth prediction. We then try out a method to incorporate utterance-based fusion of visual and lexical analysis, using string based matching.

[1]  Adam Joinson,et al.  Explanations for the Perpetration of and Reactions to Deception in a Virtual Community , 2002 .

[2]  Björn W. Schuller,et al.  Recent developments in openSMILE, the munich open-source multimedia feature extractor , 2013, ACM Multimedia.

[3]  Jay F. Nunamaker,et al.  Training Professionals to Detect Deception , 2003, ISI.

[4]  Matthew L. Jensen,et al.  HMM-Based Deception Recognition from Visual Cues , 2005, 2005 IEEE International Conference on Multimedia and Expo.

[5]  Chng Eng Siong,et al.  Modelling Public Sentiment in Twitter: Using Linguistic Patterns to Enhance Supervised Learning , 2015, CICLing.

[6]  Matthew L. Jensen,et al.  Blob Analysis of the Head and Hands: A Method for Deception Detection , 2005, Proceedings of the 38th Annual Hawaii International Conference on System Sciences.

[7]  Peter Robinson,et al.  Cross-dataset learning and person-specific normalisation for automatic Action Unit detection , 2015, 2015 11th IEEE International Conference and Workshops on Automatic Face and Gesture Recognition (FG).

[8]  Frank Rudzicz,et al.  Automatic detection of deception in child-produced speech using syntactic complexity features , 2013, ACL.

[9]  L. Fleischer.  Telling Lies Clues To Deceit In The Marketplace Politics And Marriage , 2016 .

[10]  Verónica Pérez-Rosas,et al.  A Multimodal Dataset for Deception Detection , 2014, LREC.

[11]  Erik Cambria,et al.  Sentic Computing: Exploitation of Common Sense for the Development of Emotion-Sensitive Systems , 2009, COST 2102 Training School.

[12]  Erik Cambria,et al.  SeNTU: Sentiment Analysis of Tweets by Combining a Rule-based Classifier with Supervised Learning , 2015, *SEMEVAL.

[13]  Erik Cambria,et al.  Sentic Medoids: Organizing Affective Common Sense Knowledge in a Multi-Dimensional Vector Space , 2011, ISNN.

[14]  Erik Cambria,et al.  Common Sense Knowledge Based Personality Recognition from Text , 2013, MICAI.

[15]  Venu Govindaraju,et al.  Real-time Automatic Deceit Detection from Involuntary Facial Expressions , 2007, 2007 IEEE Conference on Computer Vision and Pattern Recognition.

[16]  Erik Cambria,et al.  Towards an intelligent framework for multimodal affective data analysis , 2015, Neural Networks.

[17]  Kevin A. Johnson,et al.  Detecting Deception Using Functional Magnetic Resonance Imaging , 2005, Biological Psychiatry.

[18]  Erik Cambria,et al.  AffectiveSpace 2: Enabling Affective Intuition for Concept-Level Sentiment Analysis , 2015, AAAI.

[19]  Erik Cambria,et al.  Convolutional MKL Based Multimodal Emotion Recognition and Sentiment Analysis , 2016, 2016 IEEE 16th International Conference on Data Mining (ICDM).

[20]  Christopher Potts,et al.  Learning Word Vectors for Sentiment Analysis , 2011, ACL.

[21]  Erik Cambria,et al.  Sentic Activation: A Two-Level Affective Common Sense Reasoning Framework , 2012, AAAI.

[22]  Peter Robinson,et al.  OpenFace: An open source facial behavior analysis toolkit , 2016, 2016 IEEE Winter Conference on Applications of Computer Vision (WACV).

[23]  Matthew L. Jensen,et al.  Deception detection through automatic, unobtrusive analysis of nonverbal behavior , 2005, IEEE Intelligent Systems.

[24]  Erik Cambria,et al.  Affective Computing and Sentiment Analysis , 2016, IEEE Intelligent Systems.

[25]  S. Kosslyn,et al.  Neural correlates of different types of deception: an fMRI investigation. , 2003, Cerebral cortex.

[26]  Lawrence R. Rabiner,et al.  A tutorial on hidden Markov models and selected applications in speech recognition , 1989, Proc. IEEE.

[27]  Massimo Poesio,et al.  Automatic deception detection in Italian court cases , 2013, Artificial Intelligence and Law.

[28]  Takeo Kanade,et al.  Facial Expression Analysis , 2011, AMFG.

[29]  Ray Johnson,et al.  The contribution of executive processes to deceptive responding , 2004, Neuropsychologia.

[30]  Erik Cambria,et al.  Common Sense Computing: From the Society of Mind to Digital Intuition and beyond , 2009, COST 2101/2102 Conference.

[31]  Erik Cambria,et al.  Fusing audio, visual and textual clues for sentiment analysis from multimodal content , 2016, Neurocomputing.

[32]  R. Fisher,et al.  The cognitive interview method of conducting police interviews: eliciting extensive information and promoting therapeutic jurisprudence. , 2010, International journal of law and psychiatry.

[33]  Erik Cambria,et al.  A graph-based approach to commonsense concept extraction and semantic similarity detection , 2013, WWW.

[34]  Björn W. Schuller,et al.  SenticNet 4: A Semantic Resource for Sentiment Analysis Based on Conceptual Primitives , 2016, COLING.

[35]  Aldert Vrij,et al.  Um … they were wearing …: The effect of deception on specific hand gestures , 2012 .

[36]  Verónica Pérez-Rosas,et al.  Utterance-Level Multimodal Sentiment Analysis , 2013, ACL.

[37]  Dipankar Das,et al.  Enriching SenticNet Polarity Scores through Semi-Supervised Fuzzy Clustering , 2012, 2012 IEEE 12th International Conference on Data Mining Workshops.

[38]  Mohamed Abouelenien,et al.  Deception Detection using Real-life Trial Data , 2015, ICMI.

[39]  Fernando De la Torre,et al.  Facial Expression Analysis , 2011, Visual Analysis of Humans.

[40]  P. Ekman,et al.  Nonverbal Leakage and Clues to Deception †. , 1969, Psychiatry.

[41]  Mohamed Abouelenien,et al.  Deception detection using a multimodal approach , 2014, ICMI.

[42]  J. Pennebaker,et al.  Lying Words: Predicting Deception from Linguistic Styles , 2003, Personality & social psychology bulletin.

[43]  Jane Yung-jen Hsu,et al.  Sentic blending: Scalable multimodal fusion for the continuous interpretation of semantics and sentics , 2013, 2013 IEEE Symposium on Computational Intelligence for Human-like Intelligence (CIHLI).

[44]  Erik Cambria,et al.  Deep Convolutional Neural Network Textual Features and Multiple Kernel Learning for Utterance-level Multimodal Sentiment Analysis , 2015, EMNLP.

[45]  M. Owayjan,et al.  The design and development of a Lie Detection System using facial micro-expressions , 2012, 2012 2nd International Conference on Advances in Computational Tools for Engineering Applications (ACTEA).

[46]  R. Valencia-García,et al.  Seeing through Deception: A Computational Approach to Deceit Detection in Written Communication , 2012 .

[47]  Erik Cambria,et al.  Merging SenticNet and WordNet-Affect emotion lists for sentiment analysis , 2012, 2012 IEEE 11th International Conference on Signal Processing.

[48]  R. Valencia-García,et al.  Seeing through Deception: A Computational Approach to Deceit Detection in Spanish Written Communication , 2013 .