Smart Black Box 2.0: Efficient High-Bandwidth Driving Data Collection Based on Video Anomalies

Autonomous vehicles require fleet-wide data collection for continuous algorithm development and validation. The Smart Black Box (SBB) intelligent event data recorder has been proposed as a system for prioritized high-bandwidth data capture. This paper extends the SBB by applying anomaly detection and action detection methods for generalized event-of-interest (EOI) detection. An updated SBB pipeline is proposed for the real-time capture of driving video data. A video dataset is constructed to evaluate the SBB on real-world data for the first time. SBB performance is assessed by comparing the compression of normal and anomalous data and by comparing our prioritized data recording with a FIFO strategy. Results show that SBB data compression can increase the anomalous-to-normal storage ratio by ∼ 50%, while the prioritized recording strategy saves ∼ 25% fewer normal frames and ∼ 50-100% more anomalous frames than a FIFO queue. We compare the real-world dataset SBB results to a baseline SBB given ground-truth anomaly labels and conclude that improved general EOI detection methods will greatly improve SBB performance.

[1]  Ross B. Girshick,et al.  Mask R-CNN , 2017, 1703.06870.

[2]  Yu Yao,et al.  The Smart Black Box: A Value-Driven Automotive Event Data Recorder , 2018, 2018 21st International Conference on Intelligent Transportation Systems (ITSC).

[3]  Trevor Darrell,et al.  Spatio-Temporal Action Graph Networks , 2018, 2019 IEEE/CVF International Conference on Computer Vision Workshop (ICCVW).

[4]  Hampton C. Gabler,et al.  Crash Severity: A Comparison of Event Data Recorder Measurements with Accident Reconstruction Estimates , 2004 .

[5]  Ling Shao,et al.  Object-Centric Auto-Encoders and Dummy Anomalies for Abnormal Event Detection in Video , 2018, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[6]  Christos Dimitrakakis,et al.  TORCS, The Open Racing Car Simulator , 2005 .

[7]  Silvio Savarese,et al.  Social LSTM: Human Trajectory Prediction in Crowded Spaces , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[8]  Steven Reece,et al.  Using a Bayesian Model to Combine LDA Features with Crowdsourced Responses , 2012, TREC.

[9]  Luc Van Gool,et al.  Temporal Segment Networks: Towards Good Practices for Deep Action Recognition , 2016, ECCV.

[10]  Marco Pavone,et al.  Trajectron++: Multi-Agent Generative Trajectory Forecasting With Heterogeneous Data for Control , 2020, ArXiv.

[11]  Svetha Venkatesh,et al.  Learning Regularity in Skeleton Trajectories for Anomaly Detection in Videos , 2019, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[12]  Yong Haur Tay,et al.  Abnormal Event Detection in Videos using Spatiotemporal Autoencoder , 2017, ISNN.

[13]  Jianru Xue,et al.  DADA: A Large-scale Benchmark and Model for Driver Attention Prediction in Accidental Scenarios , 2019, arXiv.org.

[14]  Yu Yao,et al.  Egocentric Vision-based Future Vehicle Localization for Intelligent Driving Assistance Systems , 2018, 2019 International Conference on Robotics and Automation (ICRA).

[15]  Andreas Geiger,et al.  Vision meets robotics: The KITTI dataset , 2013, Int. J. Robotics Res..

[16]  Larry S. Davis,et al.  Temporal Recurrent Networks for Online Action Detection , 2018, 2019 IEEE/CVF International Conference on Computer Vision (ICCV).

[17]  Yu Yao,et al.  Unsupervised Traffic Accident Detection in First-Person Videos , 2019, 2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS).

[18]  Nanxiang Li,et al.  Driver behavior event detection for manual annotation by clustering of the driver physiological signals , 2016, 2016 IEEE 19th International Conference on Intelligent Transportation Systems (ITSC).

[19]  Jitendra Malik,et al.  SlowFast Networks for Video Recognition , 2018, 2019 IEEE/CVF International Conference on Computer Vision (ICCV).

[20]  Dot Hs,et al.  Analysis of Event Data Recorder Data for Vehicle Safety Improvement , 2008 .

[21]  Dietrich Paulus,et al.  Simple online and realtime tracking with a deep association metric , 2017, 2017 IEEE International Conference on Image Processing (ICIP).

[22]  Matthew Johnson-Roberson,et al.  BiTraP: Bi-Directional Pedestrian Trajectory Prediction With Multi-Modal Goal Estimation , 2021, IEEE Robotics and Automation Letters.

[23]  Wongun Choi,et al.  Near-Online Multi-target Tracking with Aggregated Local Flow Descriptor , 2015, 2015 IEEE International Conference on Computer Vision (ICCV).

[24]  Thomas Brox,et al.  FlowNet 2.0: Evolution of Optical Flow Estimation with Deep Networks , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[25]  Wei Liu,et al.  SSD: Single Shot MultiBox Detector , 2015, ECCV.

[26]  Yu Yao,et al.  When, Where, and What? A New Dataset for Anomaly Detection in Driving Videos , 2020, ArXiv.

[27]  Min Sun,et al.  Anticipating Accidents in Dashcam Videos , 2016, ACCV.

[28]  Shenghua Gao,et al.  Future Frame Prediction for Anomaly Detection - A New Baseline , 2017, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[29]  Jonghyun Choi,et al.  Learning Temporal Regularity in Video Sequences , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[30]  Yu Yao,et al.  The Smart Black Box: A Value-Driven High-Bandwidth Automotive Event Data Recorder , 2019, IEEE Transactions on Intelligent Transportation Systems.

[31]  Hao Chen,et al.  FCOS: Fully Convolutional One-Stage Object Detection , 2019, 2019 IEEE/CVF International Conference on Computer Vision (ICCV).

[32]  Thomas A. Dingus,et al.  An overview of the 100-car naturalistic study and findings , 2005 .

[33]  Ding Zhao,et al.  Accelerated Evaluation of Automated Vehicles. , 2016 .

[34]  Sebastian Ramos,et al.  The Cityscapes Dataset for Semantic Urban Scene Understanding , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[35]  Silvio Savarese,et al.  Learning to Track: Online Multi-object Tracking by Decision Making , 2015, 2015 IEEE International Conference on Computer Vision (ICCV).

[36]  Hiroaki Ishikawa,et al.  Self-Coaching System Based on Recorded Driving Data: Learning From One's Experiences , 2012, IEEE Transactions on Intelligent Transportation Systems.

[37]  Yann LeCun,et al.  A Closer Look at Spatiotemporal Convolutions for Action Recognition , 2017, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.