Machine Learning in Human-computer Nonverbal Communication

Nonverbal communication is an indispensable and omnipresent element of human behaviors which includes various ways such as human action, hand gesture, and facial expression, etc. As a basic means to express a person's attitudes, feelings, and emotions, it is essential in human's communication, and also play an important role in the multimodal interactions between human and computer system. Due to the richness, flexibility, and ambiguity existing in human's nonverbal expressions, as well as being affected by personalized behavior habits, the understanding and recognition of above human's expressions involves complex intelligent technologies such as visual analysis and situational awareness based on machine learning. This paper aims to provide a systematic summary and analysis of machine learning methods and technologies in human-computer nonverbal communication. It starts with the evolution of nonverbal communication research, goes on to present the definition and classification of nonverbal communication, proceeds to review and analyze the machine learning techniques which are frequently employed in human-computer nonverbal communication, and finally expounds the development of smart learning based on neural mechanism.

[1]  Weihui Dai,et al.  Smart Learning for Cyber Psychosocial and Physical Computation , 2018, 2018 IEEE International Conference on Progress in Informatics and Computing (PIC).

[2]  Alex Pentland,et al.  Real-Time American Sign Language Recognition Using Desk and Wearable Computer Based Video , 1998, IEEE Trans. Pattern Anal. Mach. Intell..

[3]  Polly Ulichny,et al.  The Culture Puzzle: Cross-Cultural Communication for English as a Second Language , 1987 .

[4]  C. Darwin.  The Expression of the Emotions in Man and Animals , .

[5]  S. Bhattacharya,et al.  Gesture classification with machine learning using Kinect sensor data , 2012, 2012 Third International Conference on Emerging Applications of Information Technology.

[6]  E. Hall.  The Silent Language , 1959 .

[7]  Emilio Soria Olivas,et al.  Handbook of Research on Machine Learning Applications and Trends : Algorithms , Methods , and Techniques , 2009 .

[8]  Narasimhan Sundararajan,et al.  A Fast and Accurate Online Sequential Learning Algorithm for Feedforward Networks , 2006, IEEE Transactions on Neural Networks.

[9]  Yuan Lan,et al.  A constructive enhancement for Online Sequential Extreme Learning Machine , 2009, 2009 International Joint Conference on Neural Networks.

[10]  Luis Paulo Reis,et al.  A comparison of machine learning algorithms applied to hand gesture recognition , 2012, 7th Iberian Conference on Information Systems and Technologies (CISTI 2012).

[11]  Narasimhan Sundararajan,et al.  A Metacognitive Neuro-Fuzzy Inference System (McFIS) for Sequential Classification Problems , 2013, IEEE Transactions on Fuzzy Systems.

[12]  Francesco Camastra,et al.  Machine Learning for Audio, Image and Video Analysis - Theory and Applications , 2007, Advanced Information and Knowledge Processing.

[13]  Larry A. Samovar,et al.  Communication Between Cultures , 1991 .

[14]  P. Ekman,et al.  BODY POSITION, FACIAL EXPRESSION, AND VERBAL BEHAVIOR DURING INTERVIEWS. , 1964, Journal of abnormal psychology.

[15]  Carol Kinsey Goman,et al.  The nonverbal advantage : secrets and science of body language at work , 2008 .

[16]  E. Hall,et al.  The Hidden Dimension , 1970 .

[17]  Nuno Lau,et al.  Machine Learning algorithms applied to the classification of robotic soccer formations and opponent teams , 2010, 2010 IEEE Conference on Cybernetics and Intelligent Systems.

[18]  Rosalind W. Picard.  Affective computing: (526112012-054) , 1997 .

[19]  Tin Hninn Hninn Maung,et al.  Real-Time Hand Tracking and Gesture Recognition System Using Neural Networks , 2009 .

[20]  K. Subramanian,et al.  Human Action Recognition using MetaCognitive Neuro-Fuzzy Inference System , 2012 .

[21]  P. Anderson.  Touching: The Human Significance of the Skin , 1978 .

[22]  P. Ekman,et al.  Unmasking the Face: A Guide to Recognizing Emotions From Facial Expressions , 1975 .

[23]  Matthew Brand,et al.  Discovery and Segmentation of Activities in Video , 2000, IEEE Trans. Pattern Anal. Mach. Intell..

[24]  Wesley E. Snyder,et al.  Machine Vision , 2003 .

[25]  M. Knapp,et al.  Nonverbal communication in human interaction , 1972 .

[26]  George N. Votsis,et al.  Emotion recognition in human-computer interaction , 2001, IEEE Signal Process. Mag..

[27]  David C. Hogg,et al.  Learning Variable-Length Markov Models of Behavior , 2001, Comput. Vis. Image Underst..

[28]  Kejun Wang,et al.  Video-Based Abnormal Human Behavior Recognition—A Review , 2012, IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews).

[29]  Badrinarayanan Rangarajan,et al.  Human action recognition in compressed domain using PBL-McRBFN approach , 2014, 2014 IEEE Ninth International Conference on Intelligent Sensors, Sensor Networks and Information Processing (ISSNIP).

[30]  Yann LeCun,et al.  Convolutional Learning of Spatio-temporal Features , 2010, ECCV.

[31]  C. Darwin.  The Expression of Emotion in Man and Animals , 2020 .

[32]  Frans Coenen.  DOI: 10.1017/S000000000000000 Printed in the United Kingdom Data Mining: Past, Present and Future , 2022 .

[33]  Stella Ting-Toomey,et al.  Understanding Intercultural Communication , 2004 .

[34]  Artur S. d'Avila Garcez,et al.  Learning motion-difference features using Gaussian restricted Boltzmann machines for efficient human action recognition , 2014, 2014 International Joint Conference on Neural Networks (IJCNN).

[35]  Ronald E. Riggio,et al.  Applications of nonverbal communication , 2005 .

[36]  Jane J. Stephan,et al.  Gesture Recognition for Human-Computer Interaction (HCI) , 2010, Int. J. Adv. Comp. Techn..

[37]  Mohamad Ivan Fanany,et al.  Constructive, robust and adaptive OS-ELM in human action recognition , 2014, 2014 International Conference on Industrial Automation, Information and Communications Technology.

[38]  Qiang Ji,et al.  Simultaneous Facial Feature Tracking and Facial Expression Recognition , 2013, IEEE Transactions on Image Processing.

[39]  Aaron F. Bobick,et al.  Recognition and interpretation of parametric gesture , 1998, Sixth International Conference on Computer Vision (IEEE Cat. No.98CH36271).

[40]  Jie Yu,et al.  Machine Learning for Audio, Image and Video Analysis , 2009, J. Electronic Imaging.

[41]  A. Montagu,et al.  Touching: The Human Significance of the Skin , 1971 .

[42]  Yonggwan Won,et al.  A Robust Online Sequential Extreme Learning Machine , 2007, ISNN.

[43]  Alex Pentland,et al.  Coupled hidden Markov models for complex action recognition , 1997, Proceedings of IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[44]  Barbara Caputo,et al.  Recognizing human actions: a local SVM approach , 2004, ICPR 2004.

[45]  Sundaram Suresh,et al.  A Meta-Cognitive Learning Algorithm for an Extreme Learning Machine Classifier , 2013, Cognitive Computation.

[46]  Sundaram Suresh,et al.  Human action recognition using Meta-Cognitive Neuro-Fuzzy Inference System , 2012, The 2012 International Joint Conference on Neural Networks (IJCNN).

[47]  J. Burgoon,et al.  Interpersonal Adaptation: Dyadic Interaction Patterns , 1995 .