Overcoming Occlusion in the Automotive Environment—A Review

Accurate and consistent vulnerable road user detection remains one of the most challenging perception tasks for autonomous vehicles. One of the most complex outstanding issues is partial occlusion, where a sensor has only a partial view of the target object due to a foreground object that partially obscures the target. A review of occlusion detection and handling solutions for the automotive environment is presented by this research. This article first discusses object detection by the human visual system, provides an overview of occlusion reasoning in computer vision, presents a summary of occlusion handling strategies in pedestrian, vehicle and object detection applications in the automotive environment. A selection of the remaining challenges to achieving the required level of object detection performance for safe autonomous driving are also discussed.

[1]  Qinping Zhao,et al.  Occlusion cues for image scene layering , 2013, Comput. Vis. Image Underst..

[2]  Bernt Schiele,et al.  CityPersons: A Diverse Dataset for Pedestrian Detection , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[3]  Jiri Matas,et al.  Online learning of robust object detectors during unstable tracking , 2009, 2009 IEEE 12th International Conference on Computer Vision Workshops, ICCV Workshops.

[4]  Song-Chun Zhu,et al.  Ieee Transactions on Pattern Analysis and Machine Intelligence Learning And-or Model to Represent Context and Occlusion for Car Detection and Viewpoint Estimation , 2022 .

[5]  Luc Van Gool,et al.  Integrating Object Detection with 3D Tracking Towards a Better Driver Assistance System , 2010, 2010 20th International Conference on Pattern Recognition.

[6]  Wanqing Li,et al.  Inter-occlusion reasoning for human detection based on variational mean field , 2013, Neurocomputing.

[7]  Wenze Hu,et al.  Modeling Occlusion by Discriminative AND-OR Structures , 2013, 2013 IEEE International Conference on Computer Vision.

[8]  Luc Van Gool,et al.  Handling Occlusions with Franken-Classifiers , 2013, 2013 IEEE International Conference on Computer Vision.

[9]  Silvio Savarese,et al.  Tracking the Untrackable: Learning to Track Multiple Cues with Long-Term Dependencies , 2017, 2017 IEEE International Conference on Computer Vision (ICCV).

[10]  Nikolaus Kriegeskorte,et al.  Recurrent Convolutional Neural Networks: A Better Model of Biological Object Recognition , 2017, bioRxiv.

[11]  G. Kreiman,et al.  Timing, Timing, Timing: Fast Decoding of Object Information from Intracranial Field Potentials in Human Visual Cortex , 2009, Neuron.

[12]  In So Kweon,et al.  KAIST Multi-Spectral Day/Night Data Set for Autonomous and Assisted Driving , 2018, IEEE Transactions on Intelligent Transportation Systems.

[13]  Chung-Cheng Chiu,et al.  Motorcycle Detection and Tracking System with Occlusion Segmentation , 2007, Eighth International Workshop on Image Analysis for Multimedia Interactive Services (WIAMIS '07).

[14]  Qing Han,et al.  A New Approach to Track Multiple Vehicles With the Combination of Robust Detection and Two Classifiers , 2018, IEEE Transactions on Intelligent Transportation Systems.

[15]  B. Yogameena,et al.  View Invariant Motorcycle Detection for Helmet Wear Analysis in Intelligent Traffic Surveillance , 2016, CVIP.

[16]  Jiun-In Guo,et al.  LiDAR/camera sensor fusion technology for pedestrian detection , 2017, 2017 Asia-Pacific Signal and Information Processing Association Annual Summit and Conference (APSIPA ASC).

[17]  Edwin Olson,et al.  Augmented vehicle tracking under occlusions for decision-making in autonomous driving , 2015, 2015 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS).

[18]  Keemin Sohn,et al.  Image-Based Learning to Measure Traffic Density Using a Deep Convolutional Neural Network , 2018, IEEE Transactions on Intelligent Transportation Systems.

[19]  Ki-Sang Hong,et al.  Deep network aided by guiding network for pedestrian detection , 2017, Pattern Recognit. Lett..

[20]  Stan Sclaroff,et al.  Improved Tracking of Multiple Humans with Trajectory Predcition and Occlusion Modeling , 1998 .

[21]  Kenji Terada,et al.  A framework for Human tracking using Kalman filter and fast mean shift algorithms , 2009, 2009 IEEE 12th International Conference on Computer Vision Workshops, ICCV Workshops.

[22]  Donald D. Hoffman,et al.  Salience of visual parts , 1997, Cognition.

[23]  Wei Zhang,et al.  Multilevel Framework to Detect and Handle Vehicle Occlusion , 2008, IEEE Transactions on Intelligent Transportation Systems.

[24]  M. Potter,et al.  Detecting and remembering pictures with and without visual noise. , 2008, Journal of vision.

[25]  Martin Lauer,et al.  Fast and Robust Cyclist Detection for Monocular Camera Systems , 2016 .

[26]  Jian Sun,et al.  Deep Residual Learning for Image Recognition , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[27]  Bin Yang,et al.  CRAFT Objects from Images , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[28]  Lili Huang,et al.  Real-time multi-vehicle tracking based on feature detection and color probability model , 2010, 2010 IEEE Intelligent Vehicles Symposium.

[29]  John Chiverton,et al.  Helmet presence classification with motorcycle detection and tracking , 2012 .

[30]  Jungwon Lee,et al.  Fused DNN: A Deep Neural Network Fusion Approach to Fast and Robust Pedestrian Detection , 2016, 2017 IEEE Winter Conference on Applications of Computer Vision (WACV).

[31]  Jeffrey S. Johnson,et al.  The recognition of partially visible natural objects in the presence and absence of their occluders , 2005, Vision Research.

[32]  Namil Kim,et al.  Multispectral pedestrian detection: Benchmark dataset and baseline , 2015, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[33]  Faliang Chang,et al.  Fast Traffic Sign Recognition via High-Contrast Region Extraction and Extended Sparse Representation , 2016, IEEE Transactions on Intelligent Transportation Systems.

[34]  Pietro Perona,et al.  Pedestrian Detection: An Evaluation of the State of the Art , 2012, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[35]  Christopher P. Heesy,et al.  Seeing in stereo: The ecology and evolution of primate binocular vision and stereopsis , 2009 .

[36]  Alan Yuille,et al.  Detecting Semantic Parts on Partially Occluded Objects , 2017, BMVC.

[37]  B. Schiele,et al.  Pedestrian detection: A benchmark , 2009, 2009 IEEE Conference on Computer Vision and Pattern Recognition.

[38]  Cheng-Ming Huang,et al.  Occlusion handling of visual tracking by fusing multiple visual clues , 2017, 2017 IEEE International Conference on Systems, Man, and Cybernetics (SMC).

[39]  Dariu Gavrila,et al.  Multi-cue pedestrian classification with partial occlusion handling , 2010, 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[40]  Kaiming He,et al.  Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks , 2015, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[41]  Jie Zhao,et al.  An approach based on mean shift and KALMAN filter for target tracking under occlusion , 2009, 2009 International Conference on Machine Learning and Cybernetics.

[42]  Josh H. McDermott,et al.  Psychophysics with junctions in real images. , 2010, Perception.

[43]  Zhaohua Chen,et al.  Occluded face recognition based on the improved SVM and block weighted LBP , 2011, 2011 International Conference on Image Analysis and Signal Processing.

[44]  Hongbin Zha,et al.  Probabilistic Inference for Occluded and Multiview On-road Vehicle Detection , 2016, IEEE Transactions on Intelligent Transportation Systems.

[45]  Hui Xiong,et al.  A new benchmark for vision-based cyclist detection , 2016, 2016 IEEE Intelligent Vehicles Symposium (IV).

[46]  Xiaoping Du,et al.  Have a Deep Look at Deformable Part Models for Cyclist Detection , 2017 .

[47]  David Gómez-Gutiérrez,et al.  Vehicle Detection with Occlusion Handling, Tracking, and OC-SVM Classification: A High Performance Vision-Based System , 2018, Sensors.

[48]  Chunluan Zhou,et al.  Learning to Integrate Occlusion-Specific Detectors for Heavily Occluded Pedestrian Detection , 2016, ACCV.

[49]  Leonidas J. Guibas,et al.  Frustum PointNets for 3D Object Detection from RGB-D Data , 2017, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[50]  Alexei A. Efros,et al.  Recovering Surface Layout from an Image , 2007, International Journal of Computer Vision.

[51]  Mohan M. Trivedi,et al.  Vehicle Detection by Independent Parts for Urban Driver Assistance , 2013, IEEE Transactions on Intelligent Transportation Systems.

[52]  Silvio Savarese,et al.  Data-driven 3D Voxel Patterns for object category recognition , 2015, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[53]  Alexei A. Efros,et al.  Recovering Occlusion Boundaries from a Single Image , 2007, 2007 IEEE 11th International Conference on Computer Vision.

[54]  Taxonomy and definitions for terms related to driving automation systems for on-road motor vehicles , 2022 .

[55]  Yu-Wing Tai,et al.  Accurate Single Stage Detector Using Recurrent Rolling Convolution , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[56]  Philippe Salembier,et al.  Monocular Depth Ordering Using T-Junctions and Convexity Occlusion Cues , 2013, IEEE Transactions on Image Processing.

[57]  Changsheng Xu,et al.  Partial Occlusion Handling for Visual Tracking via Robust Part Matching , 2014, 2014 IEEE Conference on Computer Vision and Pattern Recognition.

[58]  Soon Ki Jung,et al.  Tracking Noisy Targets: A Review of Recent Object Tracking Approaches , 2018, ArXiv.

[59]  Xiaogang Wang,et al.  Single-Pedestrian Detection Aided by Multi-pedestrian Detection , 2013, 2013 IEEE Conference on Computer Vision and Pattern Recognition.

[60]  Cristiano Premebida,et al.  Fusing LIDAR, camera and semantic information: A context-based approach for pedestrian detection , 2013, Int. J. Robotics Res..

[61]  David J. Jilk,et al.  Recurrent Processing during Object Recognition , 2011, Front. Psychol..

[62]  Chunluan Zhou,et al.  Multi-label Learning of Part Detectors for Heavily Occluded Pedestrian Detection , 2017, 2017 IEEE International Conference on Computer Vision (ICCV).

[63]  José Javier Anaya,et al.  Motorcycle detection for ADAS through camera and V2V Communication, a comparative analysis of two modern technologies , 2017, Expert Syst. Appl..

[64]  Tim Curran,et al.  The Limits of Feedforward Vision: Recurrent Processing Promotes Robust Object Recognition when Objects Are Degraded , 2012, Journal of Cognitive Neuroscience.

[65]  Youngwook Kim,et al.  Human Detection Using Doppler Radar Based on Physical Characteristics of Targets , 2015, IEEE Geoscience and Remote Sensing Letters.

[66]  Byung-Ryong Lee,et al.  Front-view car detection and counting with occlusion in dense traffic flow , 2015 .

[67]  Andrea Cavagna,et al.  SpaRTA - Tracking across occlusions via global partitioning of 3D clouds of points , 2019, IEEE transactions on pattern analysis and machine intelligence.

[68]  Pheng-Ann Heng,et al.  SINet: A Scale-Insensitive Convolutional Neural Network for Fast Vehicle Detection , 2018, IEEE Transactions on Intelligent Transportation Systems.

[69]  Kai Liu,et al.  Real-Time Multiple Vehicles Tracking with Occlusion Handling , 2011, 2011 Sixth International Conference on Image and Graphics.

[70]  José Eugenio Naranjo,et al.  Vulnerable Road Users Detection Using V2X Communications , 2015, 2015 IEEE 18th International Conference on Intelligent Transportation Systems.

[71]  Synh Viet-Uyen Ha,et al.  Occlusion vehicle detection algorithm in crowded scene for Traffic Surveillance System , 2017, 2017 International Conference on System Science and Engineering (ICSSE).

[72]  Luc Van Gool,et al.  The Pascal Visual Object Classes (VOC) Challenge , 2010, International Journal of Computer Vision.

[73]  Sang Hyuk Son,et al.  Detection scheme for a partially occluded pedestrian based on occluded depth in lidar–radar sensor fusion , 2017 .

[74]  Vincent Lepetit,et al.  Fast Keypoint Recognition in Ten Lines of Code , 2007, 2007 IEEE Conference on Computer Vision and Pattern Recognition.

[75]  Wilfried Philips,et al.  Occlusion Robust Symbol Level Fusion for Multiple People Tracking , 2017, VISIGRAPP.

[76]  Bastian Leibe,et al.  Tracking People and Their Objects , 2013, 2013 IEEE Conference on Computer Vision and Pattern Recognition.

[77]  Xiaogang Wang,et al.  Joint Deep Learning for Pedestrian Detection , 2013, 2013 IEEE International Conference on Computer Vision.

[78]  Thomas Serre,et al.  A quantitative theory of immediate visual recognition. , 2007, Progress in brain research.

[79]  Jesús García Herrero,et al.  Context aided pedestrian detection for danger estimation based on laser scanner and computer vision , 2014, Expert Syst. Appl..

[80]  B. Schiele,et al.  How Far are We from Solving Pedestrian Detection? , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[81]  Mohan M. Trivedi,et al.  Partially occluded vehicle recognition and tracking in 3D , 2013, 2013 IEEE Intelligent Vehicles Symposium (IV).

[82]  Ming Tang,et al.  PCN: Part and Context Information for Pedestrian Detection with CNNs , 2018, BMVC.

[83]  Jie Yin,et al.  Adaptive multi-strategy for multi-vehicle with mutual occlusion tracking , 2014, 2014 International Conference on Audio, Language and Image Processing.

[84]  Kunihiko Fukushima.  Recognition of partly occluded patterns: a neural network model , 2001, Biological Cybernetics.

[85]  Bernt Schiele,et al.  Monocular 3D scene understanding with explicit occlusion reasoning , 2011, CVPR 2011.

[86]  Xiaogang Wang,et al.  Deep Learning Strong Parts for Pedestrian Detection , 2015, 2015 IEEE International Conference on Computer Vision (ICCV).

[87]  Jana Kosecka,et al.  3D Bounding Box Estimation Using Deep Learning and Geometry , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[88]  Shih-Shinh Huang,et al.  Thermal pedestrian detection using block LBP with multi-level classifier , 2017, 2017 International Conference on Applied System Innovation (ICASI).

[89]  Bohyung Han,et al.  Learning occlusion with likelihoods for visual tracking , 2011, 2011 International Conference on Computer Vision.

[90]  Thierry Chateau,et al.  Deep MANTA: A Coarse-to-Fine Many-Task Network for Joint 2D and 3D Vehicle Analysis from Monocular Image , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[91]  Michael Felsberg,et al.  ECO: Efficient Convolution Operators for Tracking , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[92]  Geoffrey E. Hinton,et al.  ImageNet classification with deep convolutional neural networks , 2012, Commun. ACM.

[93]  Stefan Roth,et al.  MOT16: A Benchmark for Multi-Object Tracking , 2016, ArXiv.

[94]  Xiaogang Wang,et al.  A discriminative deep model for pedestrian detection with occlusion handling , 2012, 2012 IEEE Conference on Computer Vision and Pattern Recognition.

[95]  Bernt Schiele,et al.  Towards Reaching Human Performance in Pedestrian Detection , 2018, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[96]  KU MIN-YU,et al.  Visual Motorcycle Detection and Tracking Algorithms , 2008 .

[97]  Denis Fize,et al.  Speed of processing in the human visual system , 1996, Nature.

[98]  Bruno Vallet,et al.  Detection and 3D reconstruction of traffic signs from multiple view color images , 2013 .

[99]  Jiri Matas,et al.  Discriminative Correlation Filter with Channel and Spatial Reliability , 2017, CVPR.

[100]  Vethamani Se,et al.  Spatio-Temporal Approaches for Handling Occlusions Based on Object Tracking , 2017 .

[101]  Akihiro Takeuchi,et al.  On-Road Multivehicle Tracking Using Deformable Object Model and Particle Filter With Improved Likelihood Estimation , 2012, IEEE Transactions on Intelligent Transportation Systems.

[102]  Eric T. Shea-Brown,et al.  Dynamic representation of partially occluded objects in primate prefrontal and visual cortex , 2017, eLife.

[103]  Yiping Chen,et al.  Traffic Sign Occlusion Detection Using Mobile Laser Scanning Point Clouds , 2017, IEEE Transactions on Intelligent Transportation Systems.

[104]  Manish Singh.  Modal and Amodal Completion Generate Different Shapes , 2004, Psychological science.

[105]  Adolfo Guzmán-Arenas,et al.  COMPUTER RECOGNITION OF THREE-DIMENSIONAL OBJECTS IN A VISUAL SCENE , 1968 .

[106]  Shuicheng Yan,et al.  An HOG-LBP human detector with partial occlusion handling , 2009, 2009 IEEE 12th International Conference on Computer Vision.

[107]  Fan Yang,et al.  Exploit All the Layers: Fast and Accurate CNN Object Detector with Scale Dependent Pooling and Cascaded Rejection Classifiers , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[108]  Song-Chun Zhu,et al.  Integrating Context and Occlusion for Car Detection by Hierarchical And-Or Model , 2014, ECCV.

[109]  Pong C. Yuen,et al.  Recognition of occluded objects , 1992, Pattern Recognit..

[110]  Peter H. N. de With,et al.  Detection and handling of occlusion in an object detection system , 2015, Electronic Imaging.

[111]  A. Ghasemi,et al.  A real-time multiple vehicle classification and tracking system with occlusion handling , 2012, 2012 IEEE 8th International Conference on Intelligent Computer Communication and Processing.

[112]  Bernt Schiele,et al.  Detection and Tracking of Occluded People , 2014, International Journal of Computer Vision.

[113]  Cristiano Premebida,et al.  Pedestrian detection combining RGB and dense LIDAR data , 2014, 2014 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[114]  Wai Shiang Cheah,et al.  Occlusion Handling in Videos Object Tracking: A Survey , 2014 .

[115]  Xiaoming Liu,et al.  Illuminating Pedestrians via Simultaneous Detection and Segmentation , 2017, 2017 IEEE International Conference on Computer Vision (ICCV).

[116]  Kai-Tai Song,et al.  Image tracking of motorcycles and vehicles on urban roads and its application to traffic monitoring and enforcement , 2010 .

[117]  Jun Li,et al.  Robust traffic sign detection using fuzzy shape recognizer , 2009, International Symposium on Multispectral Image Processing and Pattern Recognition.

[118]  Michael Himmelsbach,et al.  Fusing vision and LIDAR - Synchronization, correction and occlusion reasoning , 2010, 2010 IEEE Intelligent Vehicles Symposium.

[119]  Hon-Cheng Wong,et al.  Multiple Object Detection by a Deformable Part-Based Model and an R-CNN , 2018, IEEE Signal Processing Letters.

[120]  Rogério Schmidt Feris,et al.  A Unified Multi-scale Deep Convolutional Neural Network for Fast Object Detection , 2016, ECCV.

[121]  Seiichi Mita,et al.  Occlusion handling using discriminative model of trained part templates and conditional random field , 2013, 2013 IEEE Intelligent Vehicles Symposium (IV).

[122]  Gloria Haro,et al.  Monocular Depth Ordering using Perceptual Occlusion Cues , 2016, VISIGRAPP.

[123]  Daphne Koller,et al.  A segmentation-aware object detection model with occlusion handling , 2011, CVPR 2011.

[124]  Andreas Geiger,et al.  Are we ready for autonomous driving? The KITTI vision benchmark suite , 2012, 2012 IEEE Conference on Computer Vision and Pattern Recognition.

[125]  Jin-Hee Lee,et al.  A Low-Complexity Scheme for Partially Occluded Pedestrian Detection Using LIDAR-RADAR Sensor Fusion , 2016, 2016 IEEE 22nd International Conference on Embedded and Real-Time Computing Systems and Applications (RTCSA).

[126]  Xian-Sheng Hua,et al.  Multi-Task Vehicle Detection With Region-of-Interest Voting , 2018, IEEE Transactions on Image Processing.

[127]  Cui-Hua Li,et al.  Unifying visual saliency with HOG feature learning for traffic sign detection , 2009, 2009 IEEE Intelligent Vehicles Symposium.

[128]  Jacques Verly,et al.  The State of the Art in Multiple Object Tracking Under Occlusion in Video Sequences , 2003 .

[129]  Bo Tang,et al.  Pedestrian protection using the integration of V2V and the Pedestrian Automatic Emergency Braking System , 2016, 2016 IEEE 19th International Conference on Intelligent Transportation Systems (ITSC).

[130]  José Eugenio Naranjo,et al.  Application of vehicle to another entity (V2X) communications for motorcycle crash avoidance , 2017, J. Intell. Transp. Syst..

[131]  Bilin Aksun Güvenç,et al.  Elastic band based pedestrian collision avoidance using V2X communication , 2017, 2017 IEEE Intelligent Vehicles Symposium (IV).

[132]  Nikolaus Kriegeskorte,et al.  Deep Supervised, but Not Unsupervised, Models May Explain IT Cortical Representation , 2014, PLoS Comput. Biol..

[133]  Bernd Heisele,et al.  Partially occluded object detection by finding the visible features and parts , 2015, 2015 IEEE International Conference on Image Processing (ICIP).

[134]  Marcel A. J. van Gerven,et al.  Deep Neural Networks Reveal a Gradient in the Complexity of Neural Representations across the Ventral Stream , 2014, The Journal of Neuroscience.

[135]  Simone Calderara,et al.  Visual Tracking: An Experimental Survey , 2014, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[136]  Silvio Savarese,et al.  Subcategory-Aware Convolutional Neural Networks for Object Proposals and Detection , 2016, 2017 IEEE Winter Conference on Applications of Computer Vision (WACV).

[137]  Joseph Lin Chu,et al.  The Recognition Of Partially Occluded Objects with Support Vector Machines, Convolutional Neural Networks and Deep Belief Networks , 2014, J. Artif. Intell. Soft Comput. Res..

[138]  Carl J. Debono,et al.  Efficient object selection using depth and texture information , 2016, 2016 Visual Communications and Image Processing (VCIP).

[139]  Timothée Masquelier,et al.  Unsupervised Learning of Visual Features through Spike Timing Dependent Plasticity , 2007, PLoS Comput. Biol..

[140]  Geoffrey E. Hinton,et al.  On deep generative models with applications to recognition , 2011, CVPR 2011.

[141]  Xiaogang Wang,et al.  Partial Occlusion Handling in Pedestrian Detection With a Deep Model , 2016, IEEE Transactions on Circuits and Systems for Video Technology.