A Multimodal Approach for Monitoring Driving Behavior and Emotions

Studies have indicated that emotions can significantly be influenced by environmental factors; these factors can also significantly influence drivers’ emotional state and, accordingly, their driving behavior. Furthermore, as the demand for autonomous vehicles is expected to significantly increase within the next decade, a proper understanding of drivers’/passengers’ emotions, behavior, and preferences will be needed in order to create an acceptable level of trust with humans. This paper proposes a novel semi-automated approach for understanding the effect of environmental factors on drivers’ emotions and behavioral changes through a naturalistic driving study. This setup includes a frontal road and facial camera, a smart watch for tracking physiological measurements, and a Controller Area Network (CAN) serial data logger. The results suggest that the driver’s affect is highly influenced by the type of road and the weather conditions, which have the potential to change driving behaviors. For instance, when the research defines emotional metrics as valence and engagement, results reveal there exist significant differences between human emotion in different weather conditions and road types. Participants’ engagement was higher in rainy and clear weather compared to cloudy weather. More-over, engagement was higher on city streets and highways compared to one-lane roads and two-lane highways.

[1]  Thomas A. Dingus,et al.  An overview of the 100-car naturalistic study and findings , 2005 .

[2]  Rainer Reisenzein,et al.  What is a definition of emotion? And are emotions mental-behavioral processes? , 2007 .

[3]  Dot Hs,et al.  The 100-Car Naturalistic Driving Study Phase II - Results of the 100-Car Field Experiment , 2006 .

[4]  Alex Fridman,et al.  Driver Gaze Region Estimation without Use of Eye Movement , 2015, IEEE Intelligent Systems.

[5]  Olga Sourina,et al.  Real-time EEG-based emotion monitoring using stable features , 2015, The Visual Computer.

[6]  Paul Green-Armytage,et al.  The influence of color on student emotion, heart rate, and performance in learning environments , 2016 .

[7]  Dot Hs,et al.  The 100 Car Naturalistic Driving Study , 2002 .

[8]  Christine L. Lisetti,et al.  Using Noninvasive Wearable Computers to Recognize Human Emotions from Physiological Signals , 2004, EURASIP J. Adv. Signal Process..

[9]  Maria E. Jabon,et al.  Facial expression analysis for predicting unsafe driving behavior , 2011, IEEE Pervasive Computing.

[10]  Zankhana H. Shah,et al.  Facial Expression Recognition: A Survey , 2014 .

[11]  F. B. Reguig,et al.  Analysis physiological signals for emotion recognition , 2011, International Workshop on Systems, Signal Processing and their Applications, WOSSPA.

[12]  Cole D Fitzpatrick,et al.  Evaluating the effect of vegetation and clear zone width on driver behavior using a driving simulator , 2016 .

[13]  Chien-Hung Lin,et al.  Heart Rate Variability Signal Features for Emotion Recognition by Using Principal Component Analysis and Support Vectors Machine , 2016, 2016 IEEE 16th International Conference on Bioinformatics and Bioengineering (BIBE).

[14]  Springer India-New Delhi,et al.  Automotive Revolution & Perspective Towards 2030 , 2016 .

[15]  Luca Citi,et al.  Revealing Real-Time Emotional Responses: a Personalized Assessment based on Heartbeat Dynamics , 2014, Scientific Reports.

[16]  N. A. Nijdam.  Mapping emotion to color , 2005 .

[17]  Roberto Cipolla,et al.  SegNet: A Deep Convolutional Encoder-Decoder Architecture for Robust Semantic Pixel-Wise Labelling , 2015, CVPR 2015.

[18]  Bin Hu,et al.  Estimation of valence of emotion using two frontal EEG channels , 2017, 2017 IEEE International Conference on Bioinformatics and Biomedicine (BIBM).

[19]  Daniel McDuff,et al.  AFFDEX SDK: A Cross-Platform Real-Time Multi-Face Expression Recognition Toolkit , 2016, CHI Extended Abstracts.

[20]  C. Izard.  Emotion theory and research: highlights, unanswered questions, and emerging issues. , 2009, Annual review of psychology.

[21]  Tsuyoshi Moriyama,et al.  Face analysis of aggressive moods in automobile driving using mutual subspace method , 2012, Proceedings of the 21st International Conference on Pattern Recognition (ICPR2012).

[22]  Karen K Dixon,et al.  Benefits and Risks of Urban Roadside Landscape: Finding a Livable, Balanced Response , 2007 .

[23]  Wanhui Wen,et al.  Emotion Recognition Based on Multi-Variant Correlation of Physiological Signals , 2014, IEEE Transactions on Affective Computing.

[24]  Xiaofei Xie,et al.  Negative or positive? The effect of emotion and mood on risky driving , 2013 .

[25]  Edward C. Sullivan,et al.  Investigation of Median Trees and Collisions on Urban and Suburban Conventional Highways in California , 2005 .

[26]  Alex Fridman,et al.  Cognitive Load Estimation in the Wild , 2018, CHI.

[27]  Roberto Cipolla,et al.  SegNet: A Deep Convolutional Encoder-Decoder Architecture for Image Segmentation , 2015, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[28]  Xiaomeng Li,et al.  Effect of traffic density on drivers’ lane change and overtaking maneuvers in freeway situation—A driving simulator–based study , 2018, Traffic injury prevention.

[29]  Eric Patterson,et al.  Emotion recognition using facial expressions with active appearance models , 2008 .

[30]  Yan Zhang,et al.  Driver fatigue recognition based on facial expression analysis using local binary patterns , 2015 .

[31]  J. ANDREW,et al.  Risi < of Vehicle-Tree Accidents and Management of Roadside Trees , .

[32]  Robert C. Solomon,et al.  What is an emotion? : classic and contemporary readings , 2003 .

[33]  P. Ekman.  An argument for basic emotions , 1992 .

[34]  Jody Rosenblatt Naderi,et al.  Landscape improvement impacts on roadside safety in Texas , 2006 .

[35]  Enzo Pasquale Scilingo,et al.  Recognizing Emotions Induced by Affective Sounds through Heart Rate Variability , 2015, IEEE Transactions on Affective Computing.

[36]  Thomas A. Dingus,et al.  The Impact of Driver Inattention on Near-Crash/Crash Risk: An Analysis Using the 100-Car Naturalistic Driving Study Data , 2006 .

[37]  Myounghoon Jeon,et al.  Effects of specific emotions on subjective judgment, driving performance, and perceived workload , 2014 .

[38]  Paul A. Viola,et al.  Rapid object detection using a boosted cascade of simple features , 2001, Proceedings of the 2001 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. CVPR 2001.

[39]  Gwen Littlewort,et al.  Drowsy Driver Detection Through Facial Movement Analysis , 2007, ICCV-HCI.

[40]  S. Koelsch.  Brain correlates of music-evoked emotions , 2014, Nature Reviews Neuroscience.

[41]  Chu Kiong Loo,et al.  PSO optimization of synergetic neural classifier for multichannel emotion recognition , 2010, 2010 Second World Congress on Nature and Biologically Inspired Computing (NaBIC).

[42]  M A Regan,et al.  Autonomous vehicles: human factors issues and future research , 2015 .

[43]  Daniel McDuff,et al.  Driver Frustration Detection from Audio and Video in the Wild , 2016, IJCAI.