Driver State and Behavior Detection Through Smart Wearables

Integrating driver, in-cabin, and outside environ-ment's contextual cues into the vehicle's decision making is the centerpiece of semi-automated vehicle safety. Multiple systems have been developed for providing context to the vehicle, which often rely on video streams capturing drivers' physical and environmental states. While video streams are a rich source of information, their ability in providing context can be challenging in certain situations, such as low illuminance environments (e.g., night driving), and they are highly privacy-intrusive. In this study, we leverage passive sensing through smartwatches for classifying elements of driving context. Specifically, through using the data collected from 15 participants in a naturalistic driving study, and by using multiple machine learning algorithms such as random forest, we classify driver's activities (e.g., using phone and eating), outside events (e.g., passing intersection and changing lane), and outside road attributes (e.g., driving in a city versus a highway) with an average F1 score of 94.55, 98.27, and 97.86 % respectively, through 10-fold cross-validation. Our results show the applicability of multimodal data retrieved through smart wearable devices in providing context in real-world driving scenarios and pave the way for a better shared autonomy and privacy-aware driving data-collection, analysis, and feedback for future autonomous vehicles.

[1]  Mohamed Taher Alrefaie,et al.  In a heart beat: Using driver's physiological changes to determine the quality of a takeover in highly automated vehicles. , 2019, Accident; analysis and prevention.

[2]  Rushil Khurana,et al.  Eyes on the Road: Detecting Phone Usage by Drivers Using On-Device Cameras , 2020, CHI.

[3]  John D Lee,et al.  Moving Into the Loop: An Investigation of Drivers’ Steering Behavior in Highly Automated Vehicles , 2019, Hum. Factors.

[4]  Guoliang Xing,et al.  SafeWatch: A Wearable Hand Motion Tracking System for Improving Driving Safety , 2017, 2017 IEEE/ACM Second International Conference on Internet-of-Things Design and Implementation (IoTDI).

[5]  Niall Twomey,et al.  A Comprehensive Study of Activity Recognition Using Accelerometers , 2018, Informatics.

[6]  Lu Feng,et al.  Toward Minimum Startle After Take-Over Request: A Preliminary Study of Physiological Data , 2020, AutomotiveUI.

[7]  Alex Fridman,et al.  Human-Centered Autonomous Vehicle Systems: Principles of Effective Shared Autonomy , 2018, ArXiv.

[8]  Daniel McDuff,et al.  Driver Frustration Detection from Audio and Video in the Wild , 2016, IJCAI.

[9]  Alina Roitberg,et al.  Open Set Driver Activity Recognition , 2020, 2020 IEEE Intelligent Vehicles Symposium (IV).

[10]  Philipp Wintersberger,et al.  Feasibility of smart wearables for driver drowsiness detection and its potential among different age groups , 2020, Int. J. Pervasive Comput. Commun..

[11]  Quan Z. Sheng,et al.  A review and categorization of techniques on device-free human activity recognition , 2020, J. Netw. Comput. Appl..

[12]  Lu Feng,et al.  DeepTake: Prediction of Driver Takeover Behavior using Multimodal Data , 2020, CHI.

[13]  Alex Fridman,et al.  Cognitive Load Estimation in the Wild , 2018, CHI.

[14]  Sriram Chellappan,et al.  Detecting Distracted Driving Using a Wrist-Worn Wearable , 2018, 2018 IEEE International Conference on Pervasive Computing and Communications Workshops (PerCom Workshops).

[15]  D. Bai,et al.  Stress and Heart Rate Variability: A Meta-Analysis and Review of the Literature , 2018, Psychiatry investigation.

[16]  Gyogwon Koo,et al.  Wearable Device-Based System to Monitor a Driver’s Stress, Fatigue, and Drowsiness , 2018, IEEE Transactions on Instrumentation and Measurement.

[17]  Stojan Trajanovski,et al.  Deep Physiological Arousal Detection in a Driving Simulator Using Wearable Sensors , 2017, 2017 IEEE International Conference on Data Mining Workshops (ICDMW).

[18]  Richard P. Martin,et al.  Toward Detection of Unsafe Driving with Wearables , 2015, WearSys@MobiSys.

[19]  Nitesh V. Chawla,et al.  SMOTE: Synthetic Minority Over-sampling Technique , 2002, J. Artif. Intell. Res..

[20]  Luke Stark,et al.  "I Don't Want Someone to Watch Me While I'm Working": Gendered Views of Facial Recognition Technology in Workplace Surveillance , 2020, J. Assoc. Inf. Sci. Technol..

[21]  João Gama,et al.  Human Activity Recognition Using Inertial Sensors in a Smartphone: An Overview , 2019, Sensors.

[22]  Gaël Varoquaux,et al.  Scikit-learn: Machine Learning in Python , 2011, J. Mach. Learn. Res..

[23]  Nabil Alshurafa,et al.  I Can't Be Myself: Effects of Wearable Cameras on the Capture of Authentic Behavior in the Wild , 2018, Proc. ACM Interact. Mob. Wearable Ubiquitous Technol..

[24]  Mehdi Boukhechba,et al.  HARMONY: A Human-Centered Multimodal Driving Study in the Wild , 2021, IEEE Access.

[25]  William J. Horrey,et al.  Recognition of Manual Driving Distraction Through Deep-Learning and Wearable Sensing , 2019 .

[26]  Imen Jegham,et al.  Vision-based human action recognition: An overview and real world challenges , 2020, Digit. Investig..

[27]  Mehdi Boukhechba,et al.  Personalized Driver State Profiles: A Naturalistic Data-Driven Study , 2020, AHFE.

[28]  Faicel Chamroukhi,et al.  Physical Human Activity Recognition Using Wearable Sensors , 2015, Sensors.

[29]  Mehdi Boukhechba,et al.  SWear: Sensing Using WEARables. Generalized Human Crowdsensing on Smartwatches , 2020, AHFE.

[30]  Mohan M. Trivedi,et al.  Head, Eye, and Hand Patterns for Driver Activity Recognition , 2014, 2014 22nd International Conference on Pattern Recognition.

[31]  Netzahualcóyotl Hernández,et al.  Literature Review on Transfer Learning for Human Activity Recognition Using Mobile and Wearable Devices with Environmental Technology , 2020, SN Computer Science.

[32]  Hongkai Chen,et al.  MagTrack: Enabling Safe Driving Monitoring with Wearable Magnetics , 2019, MobiSys.

[33]  I. Nourbakhsh,et al.  On the future of transportation in an era of automated and autonomous vehicles , 2019, Proceedings of the National Academy of Sciences.

[34]  William J Horrey,et al.  Detection of driver manual distraction via image-based hand and ear recognition. , 2020, Accident; analysis and prevention.

[35]  Haotian Cao,et al.  Driver activity recognition using spatial‐temporal graph convolutional LSTM networks with attention mechanism , 2020, IET Intelligent Transport Systems.

[36]  Andreas W. Kempa-Liehr,et al.  Time Series FeatuRe Extraction on basis of Scalable Hypothesis tests (tsfresh - A Python package) , 2018, Neurocomputing.