Resource-Constrained Machine Learning for ADAS: A Systematic Review

The advent of machine learning (ML) methods for the industry has opened new possibilities in the automotive domain, especially for Advanced Driver Assistance Systems (ADAS). These methods mainly focus on specific problems ranging from traffic sign and light recognition to pedestrian detection. In most cases, the computational resources and power budget found in ADAS systems are constrained while most machine learning methods are computationally intensive. The usual solution consists in adapting the ML models to comply with the memory and real-time (RT) requirements for inference. Some models are easily adapted to resource-constrained hardware, such as Support Vector Machines, while others, like Neural Networks, need more complex processes to fit into the desired hardware. The ADAS hardware (HW platforms) are diverse, from complex MPSoC CPUs down to classical MCUs, DPSs and application-specific FPGAs and ASICs or specific GPU platforms (such as the NVIDIA families Tegra or Jetson). Therefore, there is a tradeoff between the complexity of the ML model implemented and the selected platform that impacts the performance metrics: function results, energy consumption and speed (latency and throughput). In this paper, a survey in the form of systematic review is conducted to analyze the scope of the published research works that embed ML models into resource-constrained implementations for ADAS applications and what are the achievements regarding the ML performance, energy and speed trade-off.

[1]  Se-Jin Oh,et al.  Real-time vehicle detection using deep learning scheme on embedded system , 2017, 2017 Ninth International Conference on Ubiquitous and Future Networks (ICUFN).

[2]  Chi-Wei Lin,et al.  Design a Support Vector Machine-based Intelligent System for Vehicle Driving Safety Warning , 2008, 2008 11th International IEEE Conference on Intelligent Transportation Systems.

[3]  Altamiro Amadeu Susin,et al.  Reliability analysis on case-study traffic sign convolutional neural network on APSoC , 2018, 2018 IEEE 19th Latin-American Test Symposium (LATS).

[4]  Shouyi Yin,et al.  Fast and low-power behavior analysis on vehicles using smartphones , 2017, 2017 6th International Symposium on Next Generation Electronics (ISNE).

[5]  John Platt,et al.  Probabilistic Outputs for Support vector Machines and Comparisons to Regularized Likelihood Methods , 1999 .

[6]  Dmitry Yudin,et al.  Usage of fully convolutional network with clustering for traffic light detection , 2018, 2018 7th Mediterranean Conference on Embedded Computing (MECO).

[7]  Chih-Jen Lin,et al.  LIBSVM: A library for support vector machines , 2011, TIST.

[8]  Rita Cucchiara,et al.  Embedded recurrent network for head pose estimation in car , 2017, 2017 IEEE Intelligent Vehicles Symposium (IV).

[9]  Gang Li,et al.  Smartwatch-Based Wearable EEG System for Driver Drowsiness Detection , 2015, IEEE Sensors Journal.

[10]  Nanning Zheng,et al.  Knowledge-based Recurrent Attentive Neural Network for Small Object Detection , 2018 .

[11]  T. Sørensen,et al.  A method of establishing group of equal amplitude in plant sociobiology based on similarity of species content and its application to analyses of the vegetation on Danish commons , 1948 .

[12]  Helder Novais,et al.  Community based repository for georeferenced traffic signs , 2017, 2017 24º Encontro Português de Computação Gráfica e Interação (EPCGI).

[13]  Liam Kilmartin,et al.  Intra-Vehicle Networks: A Review , 2015, IEEE Transactions on Intelligent Transportation Systems.

[14]  Carlo Meghini,et al.  Deep learning for decentralized parking lot occupancy detection , 2017, Expert Syst. Appl..

[15]  Rick Salay,et al.  An Analysis of ISO 26262: Using Machine Learning Safely in Automotive Software , 2017, ArXiv.

[16]  Adam Ziebinski,et al.  Review of advanced driver assistance systems (ADAS) , 2017 .

[17]  Ana L. N. Fred,et al.  CardioWheel: ECG Biometrics on the Steering Wheel , 2015, ECML/PKDD.

[18]  Ye-Hoon Kim,et al.  Real-Time Driver Drowsiness Detection for Embedded System Using Model Compression of Deep Neural Networks , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[19]  María Jesús López Boada,et al.  Real-Time Vehicle Roll Angle Estimation Based on Neural Networks in IoT Low-Cost Devices , 2018, Sensors.

[20]  Xinming Huang,et al.  A system-on-chip FPGA design for real-time traffic signal recognition system , 2016, 2016 IEEE International Symposium on Circuits and Systems (ISCAS).

[21]  Alastair R. Allen,et al.  Using self-organising maps in the detection and recognition of road signs , 2009, Image Vis. Comput..

[22]  Christophe Cudel,et al.  Triangular traffic signs detection based on RSLD algorithm , 2013, Machine Vision and Applications.

[23]  Hyunki Kim,et al.  14.2 A 502GOPS and 0.984mW dual-mode ADAS SoC with RNN-FIS engine for intention prediction in automotive black-box system , 2016, 2016 IEEE International Solid-State Circuits Conference (ISSCC).

[24]  Zhengyou Zhang,et al.  Microsoft Kinect Sensor and Its Effect , 2012, IEEE Multim..

[25]  Nikos Komodakis,et al.  Wide Residual Networks , 2016, BMVC.

[26]  Luca Benini,et al.  Origami: A 803-GOp/s/W Convolutional Network Accelerator , 2015, IEEE Transactions on Circuits and Systems for Video Technology.

[27]  Kang Kim,et al.  Simultaneous Traffic Sign Detection and Boundary Estimation Using Convolutional Neural Network , 2018, IEEE Transactions on Intelligent Transportation Systems.

[28]  Arcady A. Putilov,et al.  Construction and validation of the EEG analogues of the Karolinska sleepiness scale based on the Karolinska drowsiness test , 2013, Clinical Neurophysiology.

[29]  Kumar Rajamani,et al.  Real-Time Vision Based Driver Drowsiness Detection Using Partial Least Squares Analysis , 2016, J. Signal Process. Syst..

[30]  Qianlong Wang,et al.  Intelligent Driving Data Recorder in Smartphone Using Deep Neural Network-Based Speedometer and Scene Understanding , 2019, IEEE Sensors Journal.

[31]  Yongbon Koo,et al.  OpenCL-Darknet: An OpenCL Implementation for Object Detection , 2018, 2018 IEEE International Conference on Big Data and Smart Computing (BigComp).

[32]  Kaiming He,et al.  Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks , 2015, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[33]  Ben Ahmed Mohamed,et al.  Detection of driver drowsiness based on the Viola & Jones method and logistic regression analysis , 2017, SCAMS '17.

[34]  Amar Ramdane-Cherif,et al.  Machine Learning Techniques in ADAS: A Review , 2018, 2018 International Conference on Advances in Computing and Communication Engineering (ICACCE).

[35]  Yunju Baek,et al.  Design and Implementation of the Vehicular Camera System using Deep Neural Network Compression , 2017, EMDL '17.

[36]  Sotiris B. Kotsiantis,et al.  Supervised Machine Learning: A Review of Classification Techniques , 2007, Informatica.

[37]  Ching-Te Chiu,et al.  Resource Efficient Hardware Implementation for Real-Time Traffic Sign Recognition , 2018, 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[38]  Yasunari Obuchi,et al.  Compact and robust speech recognition for embedded use on microprocessors , 2002, 2002 IEEE Workshop on Multimedia Signal Processing..

[39]  Kazuaki Terashima,et al.  A survey of technical trend of ADAS and autonomous driving , 2014, Proceedings of Technical Program - 2014 International Symposium on VLSI Technology, Systems and Application (VLSI-TSA).

[40]  Soojin Kim,et al.  Design of high-speed support vector machine circuit for driver assistance system , 2012, 2012 International SoC Design Conference (ISOCC).

[41]  Hakil Kim,et al.  Domain-specific data augmentation for on-road object detection based on a deep neural network , 2017, 2017 IEEE Intelligent Vehicles Symposium (IV).

[42]  A. Ghio,et al.  A Support Vector Machine based pedestrian recognition system on resource-limited hardware architectures , 2007, 2007 Ph.D Research in Microelectronics and Electronics Conference.

[43]  Beomjoon Kim,et al.  Wearable Mobile-Based Emotional Response-Monitoring System for Drivers , 2017, IEEE Transactions on Human-Machine Systems.

[44]  Forrest N. Iandola,et al.  SqueezeDet: Unified, Small, Low Power Fully Convolutional Neural Networks for Real-Time Object Detection for Autonomous Driving , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[45]  Manu Mathew,et al.  Low cost and power CNN/deep learning solution for automated driving , 2018, 2018 19th International Symposium on Quality Electronic Design (ISQED).

[46]  Jiun-In Guo,et al.  Embedded multiple object detection based on deep learning technique for advanced driver assistance system , 2017, 2017 IEEE 60th International Midwest Symposium on Circuits and Systems (MWSCAS).

[47]  Chi-Yi Tsai,et al.  An efficient automatic traffic sign detection and recognition method for smartphones , 2017, 2017 10th International Congress on Image and Signal Processing, BioMedical Engineering and Informatics (CISP-BMEI).

[48]  Germán Ros,et al.  Training Constrained Deconvolutional Networks for Road Scene Semantic Segmentation , 2016, ArXiv.

[49]  Ali Farhadi,et al.  You Only Look Once: Unified, Real-Time Object Detection , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[50]  E. Culurciello,et al.  NeuFlow: Dataflow vision processing system-on-a-chip , 2012, 2012 IEEE 55th International Midwest Symposium on Circuits and Systems (MWSCAS).

[51]  Stefano Tubaro,et al.  Reduced memory region based deep Convolutional Neural Network detection , 2016, 2016 IEEE 6th International Conference on Consumer Electronics - Berlin (ICCE-Berlin).

[52]  Manu Mathew,et al.  Efficient object detection and classification on low power embedded systems , 2017, 2017 IEEE International Conference on Consumer Electronics (ICCE).

[53]  Lawrence D. Jackel,et al.  Backpropagation Applied to Handwritten Zip Code Recognition , 1989, Neural Computation.

[54]  Heng Yang,et al.  Training a Binary Weight Object Detector by Knowledge Transfer for Autonomous Driving , 2018, 2019 International Conference on Robotics and Automation (ICRA).

[55]  David J. Kriegman,et al.  Acquiring linear subspaces for face recognition under variable lighting , 2005, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[56]  Thomas Brox,et al.  U-Net: Convolutional Networks for Biomedical Image Segmentation , 2015, MICCAI.

[57]  Oğuzhan Urhan,et al.  Real-time illumination and shadow invariant lane detection on mobile platform , 2019, Journal of Real-Time Image Processing.

[58]  Shengcai Liao,et al.  Learning Multi-scale Block Local Binary Patterns for Face Recognition , 2007, ICB.

[59]  Yu Qiao,et al.  Joint Face Detection and Alignment Using Multitask Cascaded Convolutional Networks , 2016, IEEE Signal Processing Letters.

[60]  Chin-Teng Lin,et al.  Single channel wireless EEG device for real-time fatigue level detection , 2015, 2015 International Joint Conference on Neural Networks (IJCNN).

[61]  Adam Ziebinski,et al.  A Survey of ADAS Technologies for the Future Perspective of Sensor Fusion , 2016, ICCCI.

[62]  Bo Chen,et al.  Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inference , 2017, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[63]  Naveen Kumar,et al.  An Edge Based Smart Parking Solution Using Camera Networks and Deep Learning , 2018, 2018 IEEE International Conference on Cognitive Computing (ICCC).

[64]  Chun-Hsiang Chuang,et al.  Wireless and Wearable EEG System for Evaluating Driver Vigilance , 2014, IEEE Transactions on Biomedical Circuits and Systems.

[65]  Faiyaz Doctor,et al.  A real-time driver identification system based on artificial neural networks and cepstral analysis , 2014, 2014 International Joint Conference on Neural Networks (IJCNN).

[66]  Bertrand A. Maher,et al.  Glow: Graph Lowering Compiler Techniques for Neural Networks , 2018, ArXiv.

[67]  Chi-Yi Tsai,et al.  Real-time embedded implementation of robust speed-limit sign recognition using a novel centroid-to-contour description method , 2017, IET Comput. Vis..

[68]  Minglu Li,et al.  Leveraging Audio Signals for Early Recognition of Inattentive Driving with Smartphones , 2018, IEEE Transactions on Mobile Computing.

[69]  João Paulo Papa,et al.  Embedded real-time speed limit sign recognition using image processing and machine learning techniques , 2016, Neural Computing and Applications.

[70]  Xinming Huang,et al.  RoadNet: An 80-mW Hardware Accelerator for Road Detection , 2019, IEEE Embedded Systems Letters.

[71]  Tomasz Kryjak,et al.  Embedded vision system for pedestrian detection based on HOG+SVM and use of motion information implemented in Zynq heterogeneous device , 2017, 2017 Signal Processing: Algorithms, Architectures, Arrangements, and Applications (SPA).

[72]  Dongkyu Lee,et al.  The Real-time Implementation for the Parking Line Departure Warning System , 2018, 2018 3rd IEEE International Conference on Intelligent Transportation Engineering (ICITE).

[73]  J. Joseph Antony Vision Based Vehicle Detection : A Literature Review , 2016 .

[74]  Liang-Gee Chen,et al.  A 69mW 140-meter/60fps and 60-meter/300fps intelligent vision SoC for versatile automotive applications , 2012, 2012 Symposium on VLSI Circuits (VLSIC).

[75]  Christian Maag,et al.  Enhancing Crowd Evacuation and Traffic Management Through AmI Technologies: A Review of the Literature , 2013 .

[76]  Rita Cucchiara,et al.  From Depth Data to Head Pose Estimation: A Siamese Approach , 2017, VISIGRAPP.

[77]  Stefano Tubaro,et al.  Deep Convolutional Neural Networks for pedestrian detection , 2015, Signal Process. Image Commun..

[78]  Rolf Schlagenhaft,et al.  Diagnostic Mechanism and Robustness of Safety Relevant Automotive Deep Convolutional Networks , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[79]  Zhiqiang Shen,et al.  DSOD: Learning Deeply Supervised Object Detectors from Scratch , 2017, 2017 IEEE International Conference on Computer Vision (ICCV).

[80]  Chi-Yi Tsai,et al.  A novel translation, rotation, and scale-invariant shape description method for real-time speed-limit sign recognition , 2016, 2016 International Conference on Advanced Materials for Science and Engineering (ICAMSE).

[81]  Yuan Yu,et al.  TensorFlow: A system for large-scale machine learning , 2016, OSDI.

[82]  Andrew Zisserman,et al.  Very Deep Convolutional Networks for Large-Scale Image Recognition , 2014, ICLR.

[83]  Salina Abdul Samad,et al.  Comparative Survey on Traffic Sign Detection and Recognition: a Review , 2015 .

[84]  Geoffrey E. Hinton,et al.  ImageNet classification with deep convolutional neural networks , 2012, Commun. ACM.

[85]  Pierre Charbonnier,et al.  Road Sign Detection in Images: A Case Study , 2010, 2010 20th International Conference on Pattern Recognition.

[86]  Thomas G. Dietterich Machine Learning for Sequential Data: A Review , 2002, SSPR/SPR.

[87]  Jiri Matas,et al.  Robust wide-baseline stereo from maximally stable extremal regions , 2004, Image Vis. Comput..

[88]  John Salvatier,et al.  Theano: A Python framework for fast computation of mathematical expressions , 2016, ArXiv.

[89]  Dong Wang,et al.  PipeCNN: An OpenCL-based open-source FPGA accelerator for convolution neural networks , 2017, 2017 International Conference on Field Programmable Technology (ICFPT).

[90]  Hui Wang,et al.  Pedestrian detection based on Visconti2 7502 , 2017, 2017 IEEE 17th International Conference on Communication Technology (ICCT).

[91]  Borko Furht,et al.  Motion estimation algorithms for video compression , 1996 .

[92]  Michael Botsch,et al.  Maneuver segmentation for autonomous parking based on ensemble learning , 2015, 2015 International Joint Conference on Neural Networks (IJCNN).

[93]  Ryan P. Adams,et al.  SpArSe: Sparse Architecture Search for CNNs on Resource-Constrained Microcontrollers , 2019, NeurIPS.

[94]  Feng Gao,et al.  A comprehensive review of the development of adaptive cruise control systems , 2010 .

[95]  Tajana Simunic,et al.  ORCHARD: Visual object recognition accelerator based on approximate in-memory processing , 2017, 2017 IEEE/ACM International Conference on Computer-Aided Design (ICCAD).

[96]  Alex Krizhevsky,et al.  Learning Multiple Layers of Features from Tiny Images , 2009 .

[97]  Bing-Fei Wu,et al.  Reasoning-Based Framework for Driving Safety Monitoring Using Driving Event Recognition , 2013, IEEE Transactions on Intelligent Transportation Systems.

[98]  Abdenour Hadid,et al.  An accurate eye localization approach for smart embedded system , 2016, 2016 Sixth International Conference on Image Processing Theory, Tools and Applications (IPTA).

[99]  Ayoub Al-Hamadi,et al.  Boosted human head pose estimation using kinect camera , 2015, 2015 IEEE International Conference on Image Processing (ICIP).

[100]  Andy Harter,et al.  Parameterisation of a stochastic model for human face identification , 1994, Proceedings of 1994 IEEE Workshop on Applications of Computer Vision.

[101]  Luc Van Gool,et al.  Random Forests for Real Time 3D Face Analysis , 2012, International Journal of Computer Vision.

[102]  Alberto Ferreira de Souza,et al.  Self-Driving Cars: A Survey , 2019, Expert Syst. Appl..

[103]  Xinbing Wang,et al.  Lane-Level Vehicular Localization Utilizing Smartphones , 2016, 2016 IEEE 84th Vehicular Technology Conference (VTC-Fall).

[104]  Torsten Bertram,et al.  A Fast Multi-Task CNN for Spatial Understanding of Traffic Scenes , 2018, 2018 21st International Conference on Intelligent Transportation Systems (ITSC).

[105]  Alexander Kozlov,et al.  Development of Real-time ADAS Object Detector for Deployment on CPU , 2019, IntelliSys.

[106]  Vivienne Sze,et al.  An Energy-Efficient Hardware Implementation of HOG-Based Object Detection at 1080HD 60 fps with Multi-Scale Support , 2016, J. Signal Process. Syst..

[107]  Bo Chen,et al.  MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications , 2017, ArXiv.

[108]  C. Chew‐Graham,et al.  PICO, PICOS and SPIDER: a comparison study of specificity and sensitivity in three search tools for qualitative systematic reviews , 2014, BMC Health Services Research.

[109]  Xiaoyin Xu,et al.  SqueezeMap: Fast Pedestrian Detection on a Low-Power Automotive Processor Using Efficient Convolutional Neural Networks , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[110]  Hermann Winner,et al.  Three Decades of Driver Assistance Systems: Review and Future Perspectives , 2014, IEEE Intelligent Transportation Systems Magazine.

[111]  Edward Jones,et al.  Review of pedestrian detection techniques in automotive far-infrared video , 2015 .

[112]  Wei Liu,et al.  Real-Time Traffic Light Recognition Based on Smartphone Platforms , 2017, IEEE Transactions on Circuits and Systems for Video Technology.

[113]  Aleix M. Martinez,et al.  The AR face database , 1998 .

[114]  Dong Liu,et al.  Fully Convolutional Adaptation Networks for Semantic Segmentation , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[115]  Luc Van Gool,et al.  Depth and Appearance for Mobile Scene Analysis , 2007, 2007 IEEE 11th International Conference on Computer Vision.

[116]  Robert G. Reynolds,et al.  Using Cultural Algorithms to Improve Wearable Device Gesture Recognition Performance , 2015, 2015 IEEE Symposium Series on Computational Intelligence.

[117]  Marko Bertogna,et al.  Convolutional Neural Networks on Embedded Automotive Platforms: A Qualitative Comparison , 2018, 2018 International Conference on High Performance Computing & Simulation (HPCS).

[118]  Sebastian Ramos,et al.  The Cityscapes Dataset for Semantic Urban Scene Understanding , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[119]  Sergey Ioffe,et al.  Inception-v4, Inception-ResNet and the Impact of Residual Connections on Learning , 2016, AAAI.

[120]  Davide Anguita,et al.  A Hardware-friendly Support Vector Machine for Embedded Automotive Applications , 2007, 2007 International Joint Conference on Neural Networks.

[121]  Vidya N. Murali,et al.  DeepLanes: End-To-End Lane Position Estimation Using Deep Neural Networks , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[122]  Johannes Stallkamp,et al.  Detection of traffic signs in real-world images: The German traffic sign detection benchmark , 2013, The 2013 International Joint Conference on Neural Networks (IJCNN).

[123]  Enrique Alba,et al.  Using Variable Neighborhood Search to improve the Support Vector Machine performance in embedded automotive applications , 2008, 2008 IEEE International Joint Conference on Neural Networks (IEEE World Congress on Computational Intelligence).

[124]  Vittorio Murino,et al.  Fast FPGA-based architecture for pedestrian detection based on covariance matrices , 2011, 2011 18th IEEE International Conference on Image Processing.

[125]  Luc Van Gool,et al.  Traffic sign recognition — How far are we from the solution? , 2013, The 2013 International Joint Conference on Neural Networks (IJCNN).

[126]  Alice Caplier,et al.  Sleep Deprivation Detection for Real-Time Driver Monitoring Using Deep Learning , 2018, ICIAR.

[127]  Max Welling,et al.  Bayesian Compression for Deep Learning , 2017, NIPS.

[128]  Sheroz Khan,et al.  Vehicle Detection on Embedded Single Board Computers , 2018, 2018 7th International Conference on Computer and Communication Engineering (ICCCE).

[129]  Forrest N. Iandola,et al.  SqueezeNet: AlexNet-level accuracy with 50x fewer parameters and <1MB model size , 2016, ArXiv.

[130]  Rohit Prabhavalkar,et al.  On the Efficient Representation and Execution of Deep Acoustic Models , 2016, INTERSPEECH.

[131]  Harris Drucker,et al.  Learning algorithms for classification: A comparison on handwritten digit recognition , 1995 .

[132]  Bill Triggs,et al.  Histograms of oriented gradients for human detection , 2005, 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05).

[133]  Lee Boon-Leng,et al.  Mobile-based wearable-type of driver fatigue detection by GSR and EMG , 2015, TENCON 2015 - 2015 IEEE Region 10 Conference.