Augmenting Human Intellect: Automatic Recognition of Nonverbal Behavior with Application in Deception Detection

Humans have long sought to use technology to augment human abilities and intellect. However, technology is traditionally employed only to create speedier solutions or morerapid comprehension. A more challenging endeavor is to enable humans with technology to gain additional or enhanced comprehension that may not be possible to acquire otherwise. One such application is the use of technology to augment human abilities in detecting deception using nonverbal cues. Detecting deception is often critical, whether an individual is communicating with a close friend, negotiating a business deal, or screening individuals at a security checkpoint. The detection of deception is a challenging endeavor. A variety of studies have shown that humans have a hard time accurately discriminating deception from truth, and only do so slightly better than chance. Several deception detection methods exist; however, most of these are invasive and require a controlled environment. This dissertation presents a technological approach to detecting deception based on kinesic (i.e., movement-based) and vocalic (i.e., sounds associated with the voice) cues that is firmly grounded in deception theory and past empirical studies. This noninvasive approach overcomes some of the weaknesses of other deception detection methods as it can be used in a natural environment without cooperation from the individual of interest. The automatable approach demonstrates potential for increasing humans' ability to correctly identify those who display behaviors indicative of deception. The approach 14 was evaluated using experimental and field data. The results of repeated measures analysis of variance, linear regression and discriminant function analysis suggest that the use of such a system could augment human abilities in detecting deception by as much as 15-25%. While there are a number of technical challenges that need to be addressed before such a system could be deployed in the field, there are numerous environments where it would be potentially useful.

[1]  Aldert Vrij,et al.  Behavioral Correlates of Deception in a Simulated Police Interview , 1995 .

[2]  Judee K. Burgoon,et al.  Interpersonal deception: III. Effects of deceit on perceived communication and nonverbal behavior dynamics , 1994 .

[3]  J. Burgoon,et al.  Applying a social meaning model to relational message interpretations of conversational involvement: Comparing observer and participant perspectives , 1991 .

[4]  P. Ekman,et al.  Smiles when lying. , 1988, Journal of personality and social psychology.

[5]  S. L. Sporer,et al.  The detection of deception with the reality monitoring approach: a review of the empirical evidence , 2005 .

[6]  Min C. Shin,et al.  Does colorspace transformation make any difference on skin detection? , 2002, Sixth IEEE Workshop on Applications of Computer Vision, 2002. (WACV 2002). Proceedings..

[7]  Alper Bozkurt,et al.  NIR spectroscopy measurements of cognitive load elicited by GKT and target categorization , 2003, 36th Annual Hawaii International Conference on System Sciences, 2003. Proceedings of the.

[8]  Wolfgang Effelsberg,et al.  Automatic audio content analysis , 1997, MULTIMEDIA '96.

[9]  Don W. Stacks,et al.  An introduction to communication theory , 1991 .

[10]  P. Ekman,et al.  Nonverbal Leakage and Clues to Deception †. , 1969, Psychiatry.

[11]  B. Depaulo,et al.  Accuracy of Deception Judgments , 2006, Personality and social psychology review : an official journal of the Society for Personality and Social Psychology, Inc.

[12]  R. Birdwhistell.  Kinesics and context , 1970 .

[13]  J. Burgoon,et al.  Interpersonal Deception Theory , 1996 .

[14]  Alexander H. Waibel,et al.  Strategies for automatic segmentation of audio data , 2000, 2000 IEEE International Conference on Acoustics, Speech, and Signal Processing. Proceedings (Cat. No.00CH37100).

[15]  S. Kosslyn,et al.  Neural correlates of different types of deception: an fMRI investigation. , 2003, Cerebral cortex.

[16]  Shrikanth S. Narayanan,et al.  Combining acoustic and language information for emotion recognition , 2002, INTERSPEECH.

[17]  Alan R. Hevner,et al.  Design Science in Information Systems Research , 2004, MIS Q..

[18]  John Saunders,et al.  Real-time discrimination of broadcast speech/music , 1996, 1996 IEEE International Conference on Acoustics, Speech, and Signal Processing Conference Proceedings.

[19]  Tsuhan Chen,et al.  Audio Feature Extraction and Analysis for Scene Segmentation and Classification , 1998, J. VLSI Signal Process..

[20]  Jay F. Nunamaker,et al.  Systems Development in Information Systems Research , 1990, J. Manag. Inf. Syst..

[21]  Dariu Gavrila,et al.  The Visual Analysis of Human Movement: A Survey , 1999, Comput. Vis. Image Underst..

[22]  Francis Quek,et al.  Comparison of five color models in skin pixel classification , 1999, Proceedings International Workshop on Recognition, Analysis, and Tracking of Faces and Gestures in Real-Time Systems. In Conjunction with ICCV'99 (Cat. No.PR00378).

[23]  James J. Lindsay,et al.  Cues to deception. , 2003, Psychological bulletin.

[24]  Matthew L. Jensen,et al.  Blob Analysis of the Head and Hands: A Method for Deception Detection , 2005, Proceedings of the 38th Annual Hawaii International Conference on System Sciences.

[25]  Salvatore T. March,et al.  Design and natural science research on information technology , 1995, Decis. Support Syst..

[26]  Luigi Anolli,et al.  The Voice of Deception: Vocal Strategies of Naive and Able Liars , 1997 .

[27]  A. Villringer,et al.  Near infrared spectroscopy (NIRS): A new tool to study hemodynamic changes during activation of brain function in human adults , 1993, Neuroscience Letters.

[28]  Gerald R. Miller,et al.  Deception and arousal: Isolating the behavioral correlates of deception. , 1985 .

[29]  J.F. Nunamaker,et al.  Identification of deceptive behavioral cues extracted from video , 2005, Proceedings. 2005 IEEE Intelligent Transportation Systems, 2005..

[30]  J. Burgoon,et al.  Nonverbal Behaviors, Persuasion, and Credibility , 1990 .

[31]  Timothy R. Levine,et al.  Deception Detection Accuracy is a Predictable Linear Function of Message Veracity Base-Rate: A Formal Test of Park and Levine's Probability Model , 2006 .

[32]  Frank Dellaert,et al.  Recognizing emotion in speech , 1996, Proceeding of Fourth International Conference on Spoken Language Processing. ICSLP '96.

[33]  Albert Mehrabian,et al.  Nonverbal betrayal of feeling. , 1971 .

[34]  Rama Chellappa,et al.  Human and machine recognition of faces: a survey , 1995, Proc. IEEE.

[35]  A. Glenberg,et al.  Averting the gaze disengages the environment and facilitates remembering , 1998, Memory & cognition.

[36]  Stewart P. Blake,et al.  Managing for responsive research and development , 1978 .

[37]  Hynek Hermansky,et al.  A new speaker change detection method for two-speaker segmentation , 2002, 2002 IEEE International Conference on Acoustics, Speech, and Signal Processing.

[38]  Gershon Ben-Shakhar,et al.  The validity of psychophysiological detection of information with the Guilty Knowledge Test: a meta-analytic review. , 2003, The Journal of applied psychology.

[39]  A. Vrij.  Challenging interviewees during interviews: The potential effects on lie detection , 2006 .

[40]  Aaron E. Rosenberg,et al.  A comparative performance study of several pitch detection algorithms , 1976 .

[41]  Matthew L. Jensen,et al.  An Approach for Intent Identification by Building on Deception Detection , 2005, Proceedings of the 38th Annual Hawaii International Conference on System Sciences.

[42]  Janet Beavin Bavelas,et al.  SITUATIONS THAT LEAD TO DISQUALIFICATION , 1983 .

[43]  Robert Gifford,et al.  A lens-mapping framework for understanding the encoding and decoding of interpersonal dispositions in nonverbal behavior. , 1994 .

[44]  Judee K. Burgoon,et al.  The voice of deceit: Refining and expanding vocal cues to deception , 1997 .

[45]  John H. L. Hansen,et al.  Nonlinear feature based classification of speech under stress , 2001, IEEE Trans. Speech Audio Process..

[46]  J. Burgoon,et al.  Nonverbal cues and interpersonal judgments: Participant and observer perceptions of intimacy, dominance, composure, and formality , 1999 .

[47]  Aldert Vrij,et al.  Criteria-Based Content Analysis: An empirical test of its underlying processes , 2006 .

[48]  Allen S. Lee.  Rigor and relevance in MIS research: beyond the approach of positivism alone , 1999 .

[49]  Herman Rubin,et al.  Statistical Inference in Factor Analysis , 1956 .

[50]  Nick Campbell,et al.  Perception of affect in speech - towards an automatic processing of paralinguistic information in spoken conversation , 2004, INTERSPEECH.

[51]  Thomas J. Bruneau.  Chronemics and the Verbal-Nonverbal Interface , 1981 .

[52]  Glenn Gamst,et al.  Applied Multivariate Research: Design and Interpretation , 2005 .

[53]  C. F. Bond,et al.  The miscommunication of deception: An adaptive perspective , 1985 .

[54]  Victor L. Cestaro.  A Comparison Between Decision Accuracy Rates Obtained Using the Polygraph Instrument and the Computer Voice Stress Analyzer (CVSA) in the Absence of Jeopardy. , 1995 .

[55]  P. Ekman,et al.  Detecting deception from the body or face. , 1974 .

[56]  R. Kelly Aune,et al.  The effect of probing on deceivers and truthtellers , 1989 .

[57]  K. Scherer,et al.  Vocal cues to deception: A comparative channel approach , 1985, Journal of psycholinguistic research.

[58]  Paul A. Viola,et al.  Robust Real-Time Face Detection , 2001, International Journal of Computer Vision.

[59]  Jeho Nam,et al.  Combined audio and visual streams analysis for video sequence segmentation , 1997, 1997 IEEE International Conference on Acoustics, Speech, and Signal Processing.

[60]  Riccardo Leonardi,et al.  Audio as a support to scene change detection and characterization of video sequences , 1997, 1997 IEEE International Conference on Acoustics, Speech, and Signal Processing.

[61]  M. Sambur,et al.  Selection of acoustic features for speaker identification , 1975 .

[62]  Richard J. Mammone,et al.  Speaker recognition using neural networks and conventional classifiers , 1994, IEEE Trans. Speech Audio Process..

[63]  Ruhi Sarikaya,et al.  Subband based classification of speech under stress , 1998, Proceedings of the 1998 IEEE International Conference on Acoustics, Speech and Signal Processing, ICASSP '98 (Cat. No.98CH36181).

[64]  P. Ekman,et al.  BODY MOVEMENT AND VOICE PITCH IN DECEPTIVE INTERACTION , 1976 .

[65]  John H. L. Hansen,et al.  Analysis and compensation of speech under stress and noise for environmental robustness in speech recognition , 1996, Speech Commun..

[66]  Vladimir Vezhnevets,et al.  A Survey on Pixel-Based Skin Color Detection Techniques , 2003 .

[67]  Mark A. deTurck,et al.  The Behavioral Correlates of Sanctioned and Unsanctioned Deceptive Communication , 1998 .

[68]  R. B. Jarvis.  A Pilot Study to Determine Gender Differences in the Detection of Deception: Accuracy, Cues, and Skepticism , 2005 .

[69]  John Leslie King,et al.  Rigor and relevance: careers on the line , 1999 .

[70]  Michael G. Aamodt,et al.  Who can best catch a liar?: A meta-analysis of individual differences in detecting deception. , 2006 .

[71]  Thomas H. Davenport,et al.  Rigor vs. relevance revisited: response to Benbasat and Zmud , 1999 .

[72]  Matthew L. Jensen,et al.  HMM-Based Deception Recognition from Visual Cues , 2005, 2005 IEEE International Conference on Multimedia and Expo.

[73]  J. Greenberg.  Behavior in organizations : understanding and managing the human side of work / Jerlad Greenberg and Robert A. Baron , 2000 .

[74]  Victor L. Cestaro,et al.  Effectiveness of Detection of Deception Examinations Using the Computer Voice Stress Analyzer. , 1996 .

[75]  Edward J. Delp,et al.  Optimum color spaces for skin detection , 2001, Proceedings 2001 International Conference on Image Processing (Cat. No.01CH37205).

[76]  R. Kelly Aune,et al.  Nonverbal cues to deception among intimates, friends, and strangers , 1987 .

[77]  Matthew L. Jensen,et al.  Deception detection through automatic, unobtrusive analysis of nonverbal behavior , 2005, IEEE Intelligent Systems.

[78]  John J. Grieco,et al.  Evaluation of Voice Stress Analysis Technology , 2005, Proceedings of the 38th Annual Hawaii International Conference on System Sciences.

[79]  Matthew L. Jensen,et al.  Automatic Extraction of Deceptive Behavioral Cues from Video , 2005, ISI.

[80]  Valery A. Petrushin,et al.  EMOTION IN SPEECH: RECOGNITION AND APPLICATION TO CALL CENTERS , 1999 .

[81]  R. Baumeister,et al.  Self-regulation and depletion of limited resources: does self-control resemble a muscle? , 2000, Psychological bulletin.

[82]  Thomas Hugh Feeley,et al.  Testing the Effects of Nonverbal Behavior Training on Accuracy in Deception Detection with the Inclusion of a Bogus Training Control Group , 2005 .

[83]  Hiroshi Ishiguro,et al.  Evaluation of Prosodic and Voice Quality Features on Automatic Extraction of Paralinguistic Information , 2006, 2006 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[84]  Dariusz Galasiński.  The Language of Deception: A Discourse Analytical Study , 2000 .

[85]  H. Dan O'Hair,et al.  PLANNING AND CONTROL OF BEHAVIOR DURING DECEPTION , 1985 .

[86]  J. Burgoon,et al.  Nonverbal Communication: The Unspoken Dialogue , 1988 .

[87]  Izak Benbasat,et al.  Empirical Research in Information Systems: The Practice of Relevance , 1999, MIS Q..

[88]  Douglas C. Engelbart,et al.  Augmenting human intellect: a conceptual framework , 1962 .

[89]  Ray L. Birdwhistell,et al.  Background to kinesics. , 1983 .

[90]  D. Schacter,et al.  The cognitive neuroscience of constructive memory. , 1998, Annual review of psychology.

[91]  A. Vrij.  Detecting Lies and Deceit: The Psychology of Lying and the Implications for Professional Practice , 2000 .

[92]  David G. Stork,et al.  Pattern Classification , 1973 .

[93]  P. Ekman.  Telling lies: clues to deceit in the marketplace , 1985 .

[94]  M. Zuckerman.  Verbal and nonverbal communication of deception , 1981 .

[95]  Ray Johnson,et al.  The contribution of executive processes to deceptive responding , 2004, Neuropsychologia.

[96]  Lynda M. Applegate,et al.  Rigor and Relevance in MIS Research-Introduction , 1999 .

[97]  John H. L. Hansen,et al.  Feature analysis and neural network-based classification of speech under stress , 1996, IEEE Trans. Speech Audio Process..

[98]  Robert E. Kraut,et al.  Humans as Lie Detectors , 1980 .