Distracted driver detection by combining in-vehicle and image data using deep learning

Abstract Distracted driving is among the most important reasons for traffic accidents today. Recently, there is an increasing interest in building driver assistance systems that detect the actions of the drivers and help them drive safer. In these studies, although some distinct data types such as the physical conditions of the driver, audio and visual features, car information are used; the main data source is the images of the driver that include the face, arms, and hands taken with a camera placed inside the car. In this work, we propose to integrate sensor data into the vision-based distracted driver detection model to improve the generalization ability of the system. With this purpose, we created a new data set that includes driver images and sensor data collected from real-world drives. Then, we constructed a two-stage distracted driving detection system to detect nine distracted behaviors. In the first stage, vision-based Convolutional Neural Network (CNN) models were created by transfer learning and fine-tuning methods. In the second stage, Long-Short Term Memory-Recurrent Neural Network (LSTM-RNN) models were created using sensor and image data together. We evaluate our system by two different fusion techniques and show that integrating sensor data to image-based driver detection significantly increases the overall performance with both of the fusion techniques. We also show that the accuracy of the vision-based model increases by fine-tuning the pre-trained CNN model using a related public dataset.