Affective Driver State Monitoring for Personalized, Adaptive ADAS

We seek to improve vehicle automation by using the state of the driver to develop an adaptive assistance system. We focus on the problem of measuring the driver state under varying levels of cognitive workload using affective (i.e. emotion) sensing, including thermal facial analysis and electroencephalography (EEG). This information is then used in sensor fusion and machine learning algorithms to help predict the brake reaction time of the driver, a key input in forward collision warning systems. We demonstrate the results in a pilot study, which highlights the benefits of the personalized, adaptive reaction time estimation in collision warning alert performance. A 40–50% improvement in alert precision is observed with the adaptive approach. We conclude with improvements to further strengthen the quality of the reaction time estimation and improve alert performance.

[1]  Anders Lindgren,et al.  State of the Art Analysis: An Overview of Advanced Driver Assistance Systems (ADAS) and Possible Human Factors Issues , 2006 .

[2]  Majid Komeili,et al.  Vigilance lapse identification using sparse EEG electrode arrays , 2016, 2016 IEEE Canadian Conference on Electrical and Computer Engineering (CCECE).

[3]  Ruzena Bajcsy,et al.  Semiautonomous Vehicular Control Using Driver Modeling , 2014, IEEE Transactions on Intelligent Transportation Systems.

[4]  Ruzena Bajcsy,et al.  Improved driver modeling for human-in-the-loop vehicular control , 2015, 2015 IEEE International Conference on Robotics and Automation (ICRA).

[5]  Petros A. Ioannou,et al.  Personalized Driver/Vehicle Lane Change Models for ADAS , 2015, IEEE Transactions on Vehicular Technology.

[6]  Patrick Olivier,et al.  N-backer: An auditory n-back task with automatic scoring of spoken responses , 2011, Behavior research methods.

[7]  O. Mano,et al.  Forward collision warning with a single camera , 2004, IEEE Intelligent Vehicles Symposium, 2004.

[8]  N. Kamel,et al.  Deep in thought while driving: An EEG study on drivers’ cognitive distraction , 2014 .

[9]  Peter E. Hart,et al.  Nearest neighbor pattern classification , 1967, IEEE Trans. Inf. Theory.

[10]  Marc Garbey,et al.  Interacting with human physiology , 2007, Comput. Vis. Image Underst..

[11]  Heiko Wersing,et al.  Personalization in advanced driver assistance systems and autonomous vehicles: A review , 2017, 2017 IEEE 20th International Conference on Intelligent Transportation Systems (ITSC).

[12]  J. Dusek,et al.  Impact of Incremental Increases in Cognitive Workload on Physiological Arousal and Performance in Young Adult Drivers , 2009 .

[13]  W. Klimesch,et al.  The electrophysiological dynamics of interference during the stroop task , 2008 .

[14]  T Chau,et al.  Infrared thermal imaging as a physiological access pathway: a study of the baseline characteristics of facial skin temperatures , 2009, Physiological measurement.

[15]  Jianqiang Wang,et al.  A Forward Collision Warning Algorithm With Adaptation to Driver Behaviors , 2016, IEEE Transactions on Intelligent Transportation Systems.

[16]  Dario D. Salvucci.  Modeling Driver Behavior in a Cognitive Architecture , 2006, Hum. Factors.

[17]  Mohan M. Trivedi,et al.  Speech Emotion Analysis: Exploring the Role of Context , 2010, IEEE Transactions on Multimedia.

[18]  Leo Breiman,et al.  Random Forests , 2001, Machine Learning.

[19]  Mohan M. Trivedi,et al.  Pose invariant affect analysis using thin-plate splines , 2004, Proceedings of the 17th International Conference on Pattern Recognition, 2004. ICPR 2004..

[20]  V. Gallese,et al.  Thermal infrared imaging in psychophysiology: Potentialities and limits , 2014, Psychophysiology.

[21]  A. Hamish Jamson,et al.  Potential benefits of an adaptive forward collision warning system , 2008 .

[22]  Davis E. King,et al.  Dlib-ml: A Machine Learning Toolkit , 2009, J. Mach. Learn. Res..

[23]  Timothy L. Brown,et al.  Speech-Based Interaction with In-Vehicle Computers: The Effect of Speech-Based E-Mail on Drivers' Attention to the Roadway , 2001, Hum. Factors.

[24]  Nadine Matton,et al.  Using theta and alpha band power to assess cognitive workload in multitasking environments. , 2018, International Journal of Psychophysiology.

[25]  Ayanna M. Howard,et al.  Retrieving experience: Interactive instance-based learning methods for building robot companions , 2015, 2015 IEEE International Conference on Robotics and Automation (ICRA).

[26]  Ioannis T. Pavlidis,et al.  StressCam: non-contact measurement of users' emotional states through thermal imaging , 2005, CHI Extended Abstracts.

[27]  Daniel V. McGehee,et al.  Collision Warning Timing, Driver Distraction, and Driver Response to Imminent Rear-End Collisions in a High-Fidelity Driving Simulator , 2002, Hum. Factors.

[28]  Gaël Varoquaux,et al.  Scikit-learn: Machine Learning in Python , 2011, J. Mach. Learn. Res..

[29]  Francesco Borrelli,et al.  A Learning-Based Framework for Velocity Control in Autonomous Driving , 2016, IEEE Transactions on Automation Science and Engineering.

[30]  Christine L. Lisetti.  Affective Intelligent Car Interfaces with Emotion Recognition , 2005 .

[31]  Klaus Bengler,et al.  “Take over!” How long does it take to get the driver back into the loop? , 2013 .

[32]  Tzyy-Ping Jung,et al.  EEG-based drowsiness estimation for safety driving using independent component analysis , 2005, IEEE Transactions on Circuits and Systems I: Regular Papers.

[33]  Junqiang Xi,et al.  Learning and Inferring a Driver's Braking Action in Car-Following Scenarios , 2018, IEEE Transactions on Vehicular Technology.

[34]  Hermann Winner,et al.  Three Decades of Driver Assistance Systems: Review and Future Perspectives , 2014, IEEE Intelligent Transportation Systems Magazine.

[35]  Sandra G. Hart,et al.  Nasa-Task Load Index (NASA-TLX); 20 Years Later , 2006 .