The Smart Black Box: A Value-Driven High-Bandwidth Automotive Event Data Recorder

Autonomous vehicles require reliable and resilient sensor suites and ongoing validation through fleet-wide data collection. This paper proposes a Smart Black Box (SBB) to augment traditional low-bandwidth data logging with value-driven high-bandwidth data capture. The SBB caches short-term histories of data as buffers through a deterministic Mealy machine based on data value and similarity. Compression quality for each frame is determined by optimizing the trade-off between value and storage cost. With finite storage, prioritized data recording discards low-value buffers to make room for new data. This paper formulates SBB compression decision making as a constrained multi-objective optimization problem with novel value metrics and filtering. The SBB has been evaluated on a traffic simulator which generates trajectories containing events of interest (EOIs) and corresponding first-person view videos. SBB compression efficiency is assessed by comparing storage requirements with different compression quality levels and event capture ratios. Performance is evaluated by comparing results with a traditional first-in-first-out (FIFO) recording scheme. Deep learning performance on images recorded at different compression levels is evaluated to illustrate the reproducibility of SBB recorded data.

[1]  Xiang Cheng,et al.  5G-Enabled Cooperative Intelligent Vehicular (5GenCIV) Framework: When Benz Meets Marconi , 2017, IEEE Intelligent Systems.

[2]  Anouck Girard,et al.  A Traffic Simulation Model with Interactive Drivers and High-fidelity Car Dynamics , 2019, IFAC-PapersOnLine.

[3]  Christos Dimitrakakis,et al.  TORCS, The Open Racing Car Simulator , 2005 .

[4]  James R. Sayer,et al.  Automotive Collision Avoidance System Field Operational Test Report: Methodology and Results Appendices , 2005 .

[5]  Behzad Dariush,et al.  Dynamic Traffic Scene Classification with Space-Time Coherence , 2019, 2019 International Conference on Robotics and Automation (ICRA).

[6]  Satish Narayanasamy,et al.  BugNet: continuously recording program execution for deterministic replay debugging , 2005, 32nd International Symposium on Computer Architecture (ISCA'05).

[7]  Mohan M. Trivedi,et al.  Looking at Vehicles on the Road: A Survey of Vision-Based Vehicle Detection, Tracking, and Behavior Analysis , 2013, IEEE Transactions on Intelligent Transportation Systems.

[8]  Min Sun,et al.  Anticipating Accidents in Dashcam Videos , 2016, ACCV.

[9]  Evangelia Lampiri.  Sensor anomaly detection and recovery in a nonlinear autonomous ground vehicle model , 2017, 2017 11th Asian Control Conference (ASCC).

[10]  Kazuya Takeda,et al.  Driver-Behavior Modeling Using On-Road Driving Data: A new application for behavior signal processing , 2016, IEEE Signal Processing Magazine.

[11]  Deborah Estrin,et al.  An evaluation of multi-resolution storage for sensor networks , 2003, SenSys '03.

[12]  Sebastian Ramos,et al.  The Cityscapes Dataset for Semantic Urban Scene Understanding , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[13]  Silvio Savarese,et al.  Learning to Track: Online Multi-object Tracking by Decision Making , 2015, 2015 IEEE International Conference on Computer Vision (ICCV).

[14]  Nanxiang Li,et al.  Driver behavior event detection for manual annotation by clustering of the driver physiological signals , 2016, 2016 IEEE 19th International Conference on Intelligent Transportation Systems (ITSC).

[15]  Trevor Darrell,et al.  BDD100K: A Diverse Driving Video Database with Scalable Annotation Tooling , 2018, ArXiv.

[16]  Iasonas Kokkinos,et al.  DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFs , 2016, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[17]  Ding Zhao,et al.  TrafficNet: An open naturalistic driving scenario library , 2017, 2017 IEEE 20th International Conference on Intelligent Transportation Systems (ITSC).

[18]  Trevor Darrell,et al.  Rich Feature Hierarchies for Accurate Object Detection and Semantic Segmentation , 2013, 2014 IEEE Conference on Computer Vision and Pattern Recognition.

[19]  Kejun Wang,et al.  Video-Based Abnormal Human Behavior Recognition—A Review , 2012, IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews).

[20]  Wenchao Xu,et al.  Big Data Driven Vehicular Networks , 2018, IEEE Network.

[21]  Hiroaki Ishikawa,et al.  Self-Coaching System Based on Recorded Driving Data: Learning From One's Experiences , 2012, IEEE Transactions on Intelligent Transportation Systems.

[22]  Yu Yao,et al.  Unsupervised Traffic Accident Detection in First-Person Videos , 2019, 2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS).

[23]  Dot Hs,et al.  Analysis of Event Data Recorder Data for Vehicle Safety Improvement , 2008 .

[24]  Tadahiro Taniguchi,et al.  Visualization of Driving Behavior Based on Hidden Feature Extraction by Using Deep Learning , 2017, IEEE Transactions on Intelligent Transportation Systems.

[25]  Juan Zamorano,et al.  Argos: An Advanced In-Vehicle Data Recorder on a Massively Sensorized Vehicle for Car Driver Behavior Experimentation , 2010, IEEE Transactions on Intelligent Transportation Systems.

[26]  Kaiming He,et al.  Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks , 2015, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[27]  Dietrich Paulus,et al.  Simple online and realtime tracking with a deep association metric , 2017, 2017 IEEE International Conference on Image Processing (ICIP).

[28]  Jan-Willem Romeijn,et al.  ‘All models are wrong...’: an introduction to model uncertainty , 2012 .

[29]  Antonio Iera,et al.  5G Network Slicing for Vehicle-to-Everything Services , 2017, IEEE Wireless Communications.

[30]  Brendan Tran Morris,et al.  Observing behaviors at intersections: A review of recent studies & developments , 2015, 2015 IEEE Intelligent Vehicles Symposium (IV).

[31]  Suzanne E. Lee,et al.  A COMPREHENSIVE EXAMINATION OF NATURALISTIC LANE-CHANGES , 2004 .

[32]  Nan Li,et al.  Game Theoretic Modeling of Driver and Vehicle Interactions for Verification and Validation of Autonomous Vehicle Control Systems , 2016, IEEE Transactions on Control Systems Technology.

[33]  Roberto Cipolla,et al.  SegNet: A Deep Convolutional Encoder-Decoder Architecture for Image Segmentation , 2015, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[34]  James R. Sayer,et al.  Integrated vehicle-based safety systems field operational test final program report. , 2011 .

[35]  Thomas A. Dingus,et al.  An overview of the 100-car naturalistic study and findings , 2005 .

[36]  Ding Zhao,et al.  Accelerated Evaluation of Automated Vehicles. , 2016 .

[37]  Ross B. Girshick,et al.  Fast R-CNN , 2015, 1504.08083.

[38]  Dizan Vasquez,et al.  A survey on motion prediction and risk assessment for intelligent vehicles , 2014, ROBOMECH Journal.

[39]  Xiaogang Wang,et al.  Pedestrian Behavior Understanding and Prediction with Deep Neural Networks , 2016, ECCV.

[40]  Hampton C. Gabler,et al.  Crash Severity: A Comparison of Event Data Recorder Measurements with Accident Reconstruction Estimates , 2004 .

[41]  Wongun Choi,et al.  Near-Online Multi-target Tracking with Aggregated Local Flow Descriptor , 2015, 2015 IEEE International Conference on Computer Vision (ICCV).

[42]  Ross B. Girshick,et al.  Mask R-CNN , 2017, 1703.06870.

[43]  Rainer Stiefelhagen,et al.  Pedestrian intention recognition using Latent-dynamic Conditional Random Fields , 2015, 2015 IEEE Intelligent Vehicles Symposium (IV).

[44]  Yu Yao,et al.  Egocentric Vision-based Future Vehicle Localization for Intelligent Driving Assistance Systems , 2018, 2019 International Conference on Robotics and Automation (ICRA).

[45]  Yu Yao,et al.  The Smart Black Box: A Value-Driven Automotive Event Data Recorder , 2018, 2018 21st International Conference on Intelligent Transportation Systems (ITSC).

[46]  Trevor Darrell,et al.  Classifying Collisions with Spatio-Temporal Action Graph Networks , 2018, ArXiv.

[47]  Christos G. Cassandras,et al.  Introduction to Discrete Event Systems , 1999, The Kluwer International Series on Discrete Event Dynamic Systems.

[48]  Trevor Darrell,et al.  Fully Convolutional Networks for Semantic Segmentation , 2017, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[49]  James R. Sayer,et al.  Road Departure Crash Warning System Field Operational Test: Methodology and Results. Volume 1: Technical Report , 2006 .

[50]  Prashant J. Shenoy,et al.  Rethinking Data Management for Storage-centric Sensor Networks , 2007, CIDR.

[51]  Nathalie Japkowicz,et al.  Anomaly Detection in Automobile Control Network Data with Long Short-Term Memory Networks , 2016, 2016 IEEE International Conference on Data Science and Advanced Analytics (DSAA).

[52]  Andreas Geiger,et al.  Vision meets robotics: The KITTI dataset , 2013, Int. J. Robotics Res..

[53]  Junqiang Xi,et al.  Driving Style Analysis Using Primitive Driving Patterns With Bayesian Nonparametric Approaches , 2017, IEEE Transactions on Intelligent Transportation Systems.

[54]  Ella M. Atkins,et al.  Optimal State Estimation for Systems Driven by Jump–Diffusion Process With Application to Road Anomaly Detection , 2017, IEEE Transactions on Control Systems Technology.