Car that Knows Before You Do: Anticipating Maneuvers via Learning Temporal Driving Models

Advanced Driver Assistance Systems (ADAS) have made driving safer over the last decade. They prepare vehicles for unsafe road conditions and alert drivers if they perform a dangerous maneuver. However, many accidents are unavoidable because by the time drivers are alerted, it is already too late. Anticipating maneuvers beforehand can alert drivers before they perform the maneuver and also give ADAS more time to avoid or prepare for the danger. In this work we anticipate driving maneuvers a few seconds before they occur. For this purpose we equip a car with cameras and a computing device to capture the driving context from both inside and outside of the car. We propose an Autoregressive Input-Output HMM to model the contextual information alongwith the maneuvers. We evaluate our approach on a diverse data set with 1180 miles of natural freeway and city driving and show that we can anticipate maneuvers 3.5 seconds before they occur with over 80% F1-score in real-time.

[1]  Klaus-Dieter Kuhnert,et al.  A lane change detection approach using feature ranking with maximized predictive power , 2014, 2014 IEEE Intelligent Vehicles Symposium Proceedings.

[2]  Timothy F. Cootes,et al.  Active Appearance Models , 1998, ECCV.

[3]  Simon Baker,et al.  Active Appearance Models Revisited , 2004, International Journal of Computer Vision.

[4]  Yang Wang,et al.  A dynamic conditional random field model for object segmentation in image sequences , 2005, 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05).

[5]  Fernando De la Torre,et al.  Supervised Descent Method and Its Applications to Face Alignment , 2013, 2013 IEEE Conference on Computer Vision and Pattern Recognition.

[6]  Wolfram Burgard,et al.  Learning Motion Patterns of People for Compliant Robot Motion , 2005, Int. J. Robotics Res..

[7]  George Eastman House,et al.  Sparse Bayesian Learning and the Relevance Vector Machine , 2001 .

[8]  Dieter Fox,et al.  A spatio-temporal probabilistic model for multi-sensor object recognition , 2007, 2007 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[9]  Tomohiro Yamamura,et al.  A Driver Behavior Recognition Method Based on a Driver Model Framework , 2000 .

[10]  Paul A. Viola,et al.  Robust Real-Time Face Detection , 2001, International Journal of Computer Vision.

[11]  Reinhard Klette,et al.  Look at the Driver, Look at the Road: No Distraction! No Accident! , 2014, 2014 IEEE Conference on Computer Vision and Pattern Recognition.

[12]  Klaus C. J. Dietmayer,et al.  Continuous Driver Intention Recognition with Hidden Markov Models , 2008, 2008 11th International IEEE Conference on Intelligent Transportation Systems.

[13]  Emil Angelov.  Google Self-driving car , 2016 .

[14]  Trevor Darrell,et al.  Hidden Conditional Random Fields for Gesture Recognition , 2006, 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'06).

[15]  Ruzena Bajcsy,et al.  Safe semi-autonomous control with enhanced driver modeling , 2012, 2012 American Control Conference (ACC).

[16]  Mohan M. Trivedi,et al.  Head Pose Estimation in Computer Vision: A Survey , 2009, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[17]  Fernando De la Torre,et al.  Supervised Descent Method for Solving Nonlinear Least Squares Problems in Computer Vision , 2014, ArXiv.

[18]  Sebastian Thrun,et al.  Probabilistic robotics , 2002, CACM.

[19]  Takeo Kanade,et al.  An Iterative Image Registration Technique with an Application to Stereo Vision , 1981, IJCAI.

[20]  Yoshua Bengio,et al.  An Input Output HMM Architecture , 1994, NIPS.

[21]  Kevin P. Murphy,et al.  Machine learning - a probabilistic perspective , 2012, Adaptive computation and machine learning series.

[22]  Pablo Lardelli-Claret,et al.  The influence of passengers on the risk of the driver causing a car collision in Spain. Analysis of collisions from 1990 to 1999. , 2004, Accident; analysis and prevention.

[23]  Ming Yang,et al.  DeepFace: Closing the Gap to Human-Level Performance in Face Verification , 2014, 2014 IEEE Conference on Computer Vision and Pattern Recognition.

[24]  G. G. Stokes.  "J." , 1890, The New Yale Book of Quotations.

[25]  Alex Pentland,et al.  Graphical models for driver behavior recognition in a SmartCar , 2000, Proceedings of the IEEE Intelligent Vehicles Symposium 2000 (Cat. No.00TH8511).

[26]  Tarak Gandhi,et al.  Looking-In and Looking-Out of a Vehicle: Computer-Vision-Based Enhanced Vehicle Safety , 2007, IEEE Transactions on Intelligent Transportation Systems.

[27]  J. Shotton,et al.  Decision Forests for Classification, Regression, Density Estimation, Manifold Learning and Semi-Supervised Learning , 2011 .

[28]  Luke Fletcher,et al.  Correlating driver gaze with the road scene for driver assistance systems , 2005, Robotics Auton. Syst..

[29]  Siddhartha S. Srinivasa,et al.  Formalizing Assistive Teleoperation , 2012, Robotics: Science and Systems.

[30]  Deva Ramanan,et al.  Face detection, pose estimation, and landmark localization in the wild , 2012, 2012 IEEE Conference on Computer Vision and Pattern Recognition.

[31]  Wolfram Burgard,et al.  Feature-Based Prediction of Trajectories for Socially Compliant Navigation , 2012, Robotics: Science and Systems.

[32]  Mohan M. Trivedi,et al.  Looking-in and looking-out vision for Urban Intelligent Assistance: Estimation of driver attentive state and dynamic surround for safe merging and braking , 2014, 2014 IEEE Intelligent Vehicles Symposium Proceedings.

[33]  Shaogang Gong,et al.  Recognition of group activities using dynamic probabilistic networks , 2003, Proceedings Ninth IEEE International Conference on Computer Vision.

[34]  Christian Laugier,et al.  Risk assessment at road intersections: Comparing intention and expectation , 2012, 2012 IEEE Intelligent Vehicles Symposium.

[35]  Christoph Stiller,et al.  Driver intent inference at urban intersections using the intelligent driver model , 2012, 2012 IEEE Intelligent Vehicles Symposium.

[36]  Mathias Perrollaz,et al.  Learning-based approach for online lane change intention prediction , 2013, 2013 IEEE Intelligent Vehicles Symposium (IV).

[37]  Ruzena Bajcsy,et al.  Semiautonomous Vehicular Control Using Driver Modeling , 2014, IEEE Transactions on Intelligent Transportation Systems.

[38]  Yuan Li,et al.  High-Performance Rotation Invariant Multiview Face Detection , 2007, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[39]  Carlo Tomasi,et al.  Good features to track , 1994, 1994 Proceedings of IEEE Conference on Computer Vision and Pattern Recognition.

[40]  Anup Doshi,et al.  Lane change intent prediction for driver assistance: On-road design and evaluation , 2011, 2011 IEEE Intelligent Vehicles Symposium (IV).

[41]  Trevor Darrell,et al.  Latent-Dynamic Discriminative Models for Continuous Gesture Recognition , 2007, 2007 IEEE Conference on Computer Vision and Pattern Recognition.

[42]  L. Rabiner,et al.  An introduction to hidden Markov models , 1986, IEEE ASSP Magazine.

[43]  Corinna Cortes,et al.  Support-Vector Networks , 1995, Machine Learning.

[44]  Andrew McCallum,et al.  Conditional Random Fields: Probabilistic Models for Segmenting and Labeling Sequence Data , 2001, ICML.

[45]  Dmitry Berenson,et al.  Human-robot collaborative manipulation planning using early prediction of human motion , 2013, 2013 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[46]  Markus Enzweiler,et al.  Will this car change the lane? - Turn signal recognition in the frequency domain , 2014, 2014 IEEE Intelligent Vehicles Symposium Proceedings.

[47]  Bernhard Schölkopf,et al.  Probabilistic movement modeling for intention inference in human–robot interaction , 2013, Int. J. Robotics Res..

[48]  Hema Swetha Koppula,et al.  Anticipating Human Activities Using Object Affordances for Reactive Robotic Response , 2013, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[49]  Jiri Matas,et al.  Forward-Backward Error: Automatic Detection of Tracking Failures , 2010, 2010 20th International Conference on Pattern Recognition.

[50]  Amaury Nègre,et al.  Probabilistic Analysis of Dynamic Scenes and Collision Risks Assessment to Improve Driving Safety , 2011, IEEE Intelligent Transportation Systems Magazine.

[51]  Paul Newman,et al.  Model-free detection and tracking of dynamic objects with 2D lidar , 2015, Int. J. Robotics Res..

[52]  Hema Swetha Koppula,et al.  Learning Spatio-Temporal Structure from RGB-D Videos for Human Activity Detection and Anticipation , 2013, ICML.

[53]  Ashutosh Saxena,et al.  rCRF: Recursive Belief Estimation over CRFs in RGB-D Activity Videos , 2015, Robotics: Science and Systems.

[54]  Martial Hebert,et al.  Activity Forecasting , 2012, ECCV.

[55]  Zhengyou Zhang,et al.  A Survey of Recent Advances in Face Detection , 2010 .