Know Before You Do: Anticipating Maneuvers via Learning Temporal Driving Models

Abstract : Advanced Driver Assistance Systems (ADAS) have made driving safer over the last decade. They prepare vehicles for unsafe road conditions and alert drivers if they perform a dangerous maneuver. However, many accidents are unavoidable because by the time drivers are alerted, it is already too late. Anticipating maneuvers a few seconds beforehand can alert drivers before they perform the maneuver and also give ADAS more time to avoid or prepare for the danger. Anticipation requires modeling the driver's action space, events inside the vehicle such as their head movements, and also the outside environment. Performing this joint modeling makes anticipation a challenging problem. In this work we anticipate driving maneuvers a few seconds before they occur. For this purpose we equip a car with cameras and a computing device to capture the context from both inside and outside of the car. We represent the context with expressive features and propose an Autoregressive Input-Output HMM to model the contextual information. We evaluate our approach on a diverse data set with 1180 miles of natural freeway and city driving and show that we can anticipate maneuvers 3.5 seconds before they occur with over 80% F1-score. Our computation time during inference is under 3.6 milliseconds.

[1]  Christian Kohlschein.  An introduction to Hidden Markov Models , 2007 .

[2]  Timothy F. Cootes,et al.  Active Appearance Models , 1998, ECCV.

[3]  George Eastman House,et al.  Sparse Bayesian Learning and the Relevance Vector Machine , 2001 .

[4]  Dieter Fox,et al.  A spatio-temporal probabilistic model for multi-sensor object recognition , 2007, 2007 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[5]  Hema Swetha Koppula,et al.  Learning Spatio-Temporal Structure from RGB-D Videos for Human Activity Detection and Anticipation , 2013, ICML.

[6]  Fernando De la Torre,et al.  Supervised Descent Method for Solving Nonlinear Least Squares Problems in Computer Vision , 2014, ArXiv.

[7]  Martial Hebert,et al.  Activity Forecasting , 2012, ECCV.

[8]  Zhengyou Zhang,et al.  A Survey of Recent Advances in Face Detection , 2010 .

[9]  Deva Ramanan,et al.  Face detection, pose estimation, and landmark localization in the wild , 2012, 2012 IEEE Conference on Computer Vision and Pattern Recognition.

[10]  J. Shotton,et al.  Decision Forests for Classification, Regression, Density Estimation, Manifold Learning and Semi-Supervised Learning , 2011 .

[11]  Yuan Li,et al.  High-Performance Rotation Invariant Multiview Face Detection , 2007, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[12]  Carlo Tomasi,et al.  Good features to track , 1994, 1994 Proceedings of IEEE Conference on Computer Vision and Pattern Recognition.

[13]  Anup Doshi,et al.  Lane change intent prediction for driver assistance: On-road design and evaluation , 2011, 2011 IEEE Intelligent Vehicles Symposium (IV).

[14]  Tarak Gandhi,et al.  Looking-In and Looking-Out of a Vehicle: Computer-Vision-Based Enhanced Vehicle Safety , 2007, IEEE Transactions on Intelligent Transportation Systems.

[15]  Sebastian Thrun,et al.  Probabilistic robotics , 2002, CACM.

[16]  Takeo Kanade,et al.  An Iterative Image Registration Technique with an Application to Stereo Vision , 1981, IJCAI.

[17]  Klaus C. J. Dietmayer,et al.  Continuous Driver Intention Recognition with Hidden Markov Models , 2008, 2008 11th International IEEE Conference on Intelligent Transportation Systems.

[18]  Wolfram Burgard,et al.  Learning Motion Patterns of People for Compliant Robot Motion , 2005, Int. J. Robotics Res..

[19]  Tomohiro Yamamura,et al.  A Driver Behavior Recognition Method Based on a Driver Model Framework , 2000 .

[20]  Paul A. Viola,et al.  Robust Real-Time Face Detection , 2001, International Journal of Computer Vision.

[21]  Wolfram Burgard,et al.  Feature-Based Prediction of Trajectories for Socially Compliant Navigation , 2012, Robotics: Science and Systems.

[22]  Trevor Darrell,et al.  Latent-Dynamic Discriminative Models for Continuous Gesture Recognition , 2007, 2007 IEEE Conference on Computer Vision and Pattern Recognition.

[23]  Yoshua Bengio,et al.  An Input Output HMM Architecture , 1994, NIPS.

[24]  Kevin P. Murphy,et al.  Machine learning - a probabilistic perspective , 2012, Adaptive computation and machine learning series.

[25]  Pablo Lardelli-Claret,et al.  The influence of passengers on the risk of the driver causing a car collision in Spain. Analysis of collisions from 1990 to 1999. , 2004, Accident; analysis and prevention.

[26]  Paul Newman,et al.  Model-free detection and tracking of dynamic objects with 2D lidar , 2015, Int. J. Robotics Res..

[27]  Mohan M. Trivedi,et al.  Looking-in and looking-out vision for Urban Intelligent Assistance: Estimation of driver attentive state and dynamic surround for safe merging and braking , 2014, 2014 IEEE Intelligent Vehicles Symposium Proceedings.

[28]  Shaogang Gong,et al.  Recognition of group activities using dynamic probabilistic networks , 2003, Proceedings Ninth IEEE International Conference on Computer Vision.

[29]  Siddhartha S. Srinivasa,et al.  Formalizing Assistive Teleoperation , 2012, Robotics: Science and Systems.

[30]  Dmitry Berenson,et al.  Human-robot collaborative manipulation planning using early prediction of human motion , 2013, 2013 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[31]  Markus Enzweiler,et al.  Will this car change the lane? - Turn signal recognition in the frequency domain , 2014, 2014 IEEE Intelligent Vehicles Symposium Proceedings.

[32]  Bernhard Schölkopf,et al.  Probabilistic movement modeling for intention inference in human–robot interaction , 2013, Int. J. Robotics Res..

[33]  Hema Swetha Koppula,et al.  Anticipating Human Activities Using Object Affordances for Reactive Robotic Response , 2013, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[34]  Jiri Matas,et al.  Forward-Backward Error: Automatic Detection of Tracking Failures , 2010, 2010 20th International Conference on Pattern Recognition.

[35]  Christoph Stiller,et al.  Driver intent inference at urban intersections using the intelligent driver model , 2012, 2012 IEEE Intelligent Vehicles Symposium.

[36]  Klaus-Dieter Kuhnert,et al.  A lane change detection approach using feature ranking with maximized predictive power , 2014, 2014 IEEE Intelligent Vehicles Symposium Proceedings.

[37]  Yang Wang,et al.  A dynamic conditional random field model for object segmentation in image sequences , 2005, 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05).

[38]  Fernando De la Torre,et al.  Supervised Descent Method and Its Applications to Face Alignment , 2013, 2013 IEEE Conference on Computer Vision and Pattern Recognition.

[39]  Mohan M. Trivedi,et al.  Head Pose Estimation in Computer Vision: A Survey , 2009, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[40]  Alex Pentland,et al.  Graphical models for driver behavior recognition in a SmartCar , 2000, Proceedings of the IEEE Intelligent Vehicles Symposium 2000 (Cat. No.00TH8511).

[41]  Amaury Nègre,et al.  Probabilistic Analysis of Dynamic Scenes and Collision Risks Assessment to Improve Driving Safety , 2011, IEEE Intelligent Transportation Systems Magazine.

[42]  Trevor Darrell,et al.  Hidden Conditional Random Fields for Gesture Recognition , 2006, 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'06).

[43]  Christian Laugier,et al.  Risk assessment at road intersections: Comparing intention and expectation , 2012, 2012 IEEE Intelligent Vehicles Symposium.

[44]  Mathias Perrollaz,et al.  Learning-based approach for online lane change intention prediction , 2013, 2013 IEEE Intelligent Vehicles Symposium (IV).

[45]  P. Claret,et al.  Age and sex differences in the risk of causing vehicle collisions in Spain, 1990 to 1999. , 2003, Accident; analysis and prevention.

[46]  L. Rabiner,et al.  An introduction to hidden Markov models , 1986, IEEE ASSP Magazine.

[47]  Corinna Cortes,et al.  Support-Vector Networks , 1995, Machine Learning.

[48]  Andrew McCallum,et al.  Conditional Random Fields: Probabilistic Models for Segmenting and Labeling Sequence Data , 2001, ICML.

[49]  Ming Yang,et al.  DeepFace: Closing the Gap to Human-Level Performance in Face Verification , 2014, 2014 IEEE Conference on Computer Vision and Pattern Recognition.

[50]  Simon Baker,et al.  Active Appearance Models Revisited , 2004, International Journal of Computer Vision.

[51]  Reinhard Klette,et al.  Look at the Driver, Look at the Road: No Distraction! No Accident! , 2014, 2014 IEEE Conference on Computer Vision and Pattern Recognition.