Predicting behavioral competencies automatically from facial expressions in real-time video-recorded interviews

This work aims to develop a real-time image and video processor enabled with an artificial intelligence (AI) agent that can predict a job candidate’s behavioral competencies according to his or her facial expressions. This is accomplished using a real-time video-recorded interview with a histogram of oriented gradients and support vector machine (HOG-SVM) plus convolutional neural network (CNN) recognition. Different from the classical view of recognizing emotional states, this prototype system was developed to automatically decode a job candidate’s behaviors by their microexpressions based on the behavioral ecology view of facial displays (BECV) in the context of employment interviews using a real-time video-recorded interview. An experiment was conducted at a Fortune 500 company, and the video records and competency scores were collected from the company’s employees and hiring managers. The results indicated that our proposed system can provide better predictive power than can human-structured interviews, personality inventories, occupation interest testing, and assessment centers. As such, our proposed approach can be utilized as an effective screening method using a personal-value-based competency model.

[1]  The Usage of Grayscale or Color Images for Facial Expression Recognition with Deep Neural Networks , 2019 .

[2]  Sheng-Yi Wu,et al.  Applying data mining techniques to explore user behaviors and watching video patterns in converged IT environments , 2021, Journal of ambient intelligence and humanized computing.

[3]  Lyle M. Spencer,et al.  Competencies: The Right Foundation The Right Foundation For Effective Human Resources Management , 1996 .

[4]  Ninad Mehendale,et al.  Facial emotion recognition using convolutional neural networks (FERC) , 2020, SN Applied Sciences.

[5]  P. Ekman,et al.  Nonverbal Leakage and Clues to Deception †. , 1969, Psychiatry.

[6]  Paul Ekman,et al.  What Scientists Who Study Emotion Agree About , 2016, Perspectives on psychological science : a journal of the Association for Psychological Science.

[7]  Aleix M. Martinez,et al.  Emotional Expressions Reconsidered: Challenges to Inferring Emotion From Human Facial Movements , 2019, Psychological science in the public interest : a journal of the American Psychological Society.

[8]  Cheng Dai,et al.  Human action recognition using two-stream attention based LSTM networks , 2020, Appl. Soft Comput..

[9]  Qiang Wu,et al.  A survey: facial micro-expression recognition , 2017, Multimedia Tools and Applications.

[10]  Edilson de Aguiar,et al.  Facial expression recognition with Convolutional Neural Networks: Coping with few data and the training sample order , 2017, Pattern Recognit..

[11]  Guoying Zhao,et al.  A Main Directional Mean Optical Flow Feature for Spontaneous Micro-Expression Recognition , 2016, IEEE Transactions on Affective Computing.

[12]  José-Miguel Fernández-Dols,et al.  Are smiles a sign of happiness? Spontaneous expressions of judo winners , 2015 .

[13]  Paul A. Viola,et al.  Rapid object detection using a boosted cascade of simple features , 2001, Proceedings of the 2001 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. CVPR 2001.

[14]  Stefanos Zafeiriou,et al.  Robust Discriminative Response Map Fitting with Constrained Local Models , 2013, 2013 IEEE Conference on Computer Vision and Pattern Recognition.

[15]  Bill Triggs,et al.  Histograms of oriented gradients for human detection , 2005, 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05).

[16]  Pierluigi Carcagnì,et al.  Facial expression recognition and histograms of oriented gradients: a comprehensive study , 2015, SpringerPlus.

[17]  Mohamed Elhoseny,et al.  Deep learning model for real-time image compression in Internet of Underwater Things (IoUT) , 2020, Journal of Real-Time Image Processing.

[18]  Muhammad Sajjad,et al.  Human Behavior Understanding in Big Multimedia Data Using CNN based Facial Expression Recognition , 2019, Mob. Networks Appl..

[19]  Lyle M. Spencer,et al.  Competence at Work: Models for Superior Performance , 1993 .

[20]  Norman I. Badler,et al.  Investigating Macroexpressions and Microexpressions in Computer Graphics Animated Faces , 2014, PRESENCE: Teleoperators and Virtual Environments.

[21]  Chin-Feng Lai,et al.  A Low-Latency Object Detection Algorithm for the Edge Devices of IoV Systems , 2020, IEEE Transactions on Vehicular Technology.

[22]  T. DeGroot,et al.  Can Nonverbal Cues be Used to Make Meaningful Personality Attributions in Employment Interviews? , 2009 .

[23]  Jani Boutellier,et al.  Evaluation of real-time LBP computing in multiple architectures , 2014, Journal of Real-Time Image Processing.

[24]  Han-Chieh Chao,et al.  Human Behavior Deep Recognition Architecture for Smart City Applications in the 5G Environment , 2019, IEEE Network.

[25]  K. S. Gayathri,et al.  Facial Emotion Recognition using Convolutional Neural Networks , 2019, ArXiv.

[26]  Paul A. Viola,et al.  Robust Real-Time Face Detection , 2001, International Journal of Computer Vision.

[27]  Chien-Liang Lin,et al.  Intelligent video interview agent used to predict communication skill and perceived personality traits , 2020, Human-centric Computing and Information Sciences.

[28]  Laurence T. Yang,et al.  Video Scene Segmentation Using Tensor-Train Faster-RCNN for Multimedia IoT Systems , 2021, IEEE Internet of Things Journal.

[29]  Klaus G. Melchers,et al.  A review of applicant faking in selection interviews , 2020 .

[30]  Hugo Jair Escalante,et al.  Recognition of facial expressions based on CNN features , 2020, Multimedia Tools and Applications.

[31]  I. Nikolaou.  The Development and Validation of a Measure of Generic Work Competencies , 2003 .

[32]  Chin-Feng Lai,et al.  Bibliometric study of social network analysis literature , 2019, Libr. Hi Tech.

[33]  Chien-Hsing Chou,et al.  A Finger-Worn Device for Exploring Chinese Printed Text With Using CNN Algorithm on a Micro IoT Processor , 2019, IEEE Access.

[34]  Benjamin Johnston,et al.  A review of image-based automatic facial landmark identification techniques , 2018, EURASIP Journal on Image and Video Processing.

[35]  Bacha Rehman,et al.  Face detection and tracking using hybrid margin-based ROI techniques , 2019, The Visual Computer.

[36]  Robert L. Cardy,et al.  Competencies: Alternative frameworks for competitive advantage , 2006 .

[37]  Philip L. Roth,et al.  Understanding applicant behavior in employment interviews: A theoretical model of interviewee performance , 2011 .

[38]  Yue Zhao,et al.  A Convolutional Neural Network for Compound Micro-Expression Recognition , 2019, Sensors.

[39]  A. J. Fridlund,et al.  Facial Displays Are Tools for Social Influence , 2018, Trends in Cognitive Sciences.

[40]  Andrea Cavallaro,et al.  Predicting and recognizing human interactions in public spaces , 2014, Journal of Real-Time Image Processing.

[41]  Davis E. King,et al.  Dlib-ml: A Machine Learning Toolkit , 2009, J. Mach. Learn. Res..

[42]  Chuen-Fa Ni,et al.  Applying deep learning algorithms to enhance simulations of large-scale groundwater flow in IoTs , 2020, Appl. Soft Comput..

[43]  L. F. Barrett,et al.  Facial Expression Predictions as Drivers of Social Perception , 2018, Journal of personality and social psychology.

[44]  Oge Marques,et al.  Dropout vs. batch normalization: an empirical study of their impact to deep learning , 2020, Multimedia Tools and Applications.

[45]  Bernd Tibken,et al.  Using neural networks to enhance the quality of ROIs for video based remote heart rate measurement from human faces , 2019, 2019 IEEE International Conference on Consumer Electronics (ICCE).

[46]  M. Terpenning,et al.  Delusional parasitosis in geriatric patients. , 1991, Psychosomatics.

[47]  Chien-Liang Lin,et al.  TensorFlow-Based Automatic Personality Recognition Used in Asynchronous Video Interviews , 2019, IEEE Access.

[48]  Abdulkadir Sengur,et al.  Efficient deep features selections and classification for flower species recognition , 2019, Measurement.

[49]  Mei-I Cheng,et al.  Competence, competency and competencies: performance assessment in organisations , 2002 .

[50]  A. J. Fridlund.  Human Facial Expression: An Evolutionary View , 1994 .

[51]  Shih-Hao Lu,et al.  Does the use of synchrony and artificial intelligence in video interviews affect interview ratings and applicant attitudes? , 2019, Comput. Hum. Behav..

[52]  Qi Wu,et al.  Effects of the duration of expressions on the recognition of microexpressions , 2012, Journal of Zhejiang University SCIENCE B.

[53]  James T. Kochanski.  Introduction to special issue on human resource competencies , 2009 .

[54]  K. Taber.  The Use of Cronbach’s Alpha When Developing and Reporting Research Instruments in Science Education , 2017, Research in Science Education.

[55]  Zied Lachiri,et al.  Comparison of Haar-like, HOG and LBP approaches for face detection in video sequences , 2019, 2019 16th International Multi-Conference on Systems, Signals & Devices (SSD).

[56]  B. Waller,et al.  Rethinking primate facial expression: A predictive framework , 2017, Neuroscience & Biobehavioral Reviews.

[57]  Gerhard Rigoll,et al.  Robust Facial Landmark Detection via a Fully-Convolutional Local-Global Context Network , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[58]  Alan J. Fridlund,et al.  The behavioral ecology view of facial displays, 25 years later , 2017 .

[59]  George Trigeorgis,et al.  Joint Multi-View Face Alignment in the Wild , 2017, IEEE Transactions on Image Processing.

[60]  Richard A. Posthuma,et al.  Are we asking the right questions? Predictive validity comparison of four structured interview question types , 2019, Journal of Business Research.

[61]  Yu-Sheng Su,et al.  Social Facebook With Big Six Approaches for Improved Students’ Learning Performance and Behavior: A Case Study of a Project Innovation and Implementation Course , 2020, Frontiers in Psychology.

[62]  John See,et al.  A Survey of Automatic Facial Micro-Expression Analysis: Databases, Methods, and Challenges , 2018, Front. Psychol..

[63]  Akif Durdu,et al.  CNN and HOG based comparison study for complete occlusion handling in human tracking , 2020 .

[64]  Majid Mirmehdi,et al.  DS-KCF: a real-time tracker for RGB-D data , 2016, Journal of Real-Time Image Processing.

[65]  Kup-Sze Choi,et al.  Fast Gabor texture feature extraction with separable filters using GPU , 2013, Journal of Real-Time Image Processing.

[66]  Hamid Sadeghi,et al.  Human vision inspired feature extraction for facial expression recognition , 2019, Multimedia Tools and Applications.

[67]  Dewi Yanti Liliana,et al.  Enhancing CNN with Preprocessing Stage in Automatic Emotion Recognition , 2017, ICCSCI.

[68]  I. Z. Reguly,et al.  PPCU Sam: Open-source face recognition framework , 2019, KES.

[69]  Dmitry B. Goldgof,et al.  Macro- and micro-expression spotting in long videos using spatio-temporal strain , 2011, Face and Gesture 2011.

[70]  Zakia Hammal,et al.  Facial expression classification: An approach based on the fusion of facial deformations using the transferable belief model , 2007, Int. J. Approx. Reason..