Pedestrian Detection at Day/Night Time with Visible and FIR Cameras: A Comparison

Despite all the significant advances in pedestrian detection brought by computer vision for driving assistance, it is still a challenging problem. One reason is the extremely varying lighting conditions under which such a detector should operate, namely day and nighttime. Recent research has shown that the combination of visible and non-visible imaging modalities may increase detection accuracy, where the infrared spectrum plays a critical role. The goal of this paper is to assess the accuracy gain of different pedestrian models (holistic, part-based, patch-based) when training with images in the far infrared spectrum. Specifically, we want to compare detection accuracy on test images recorded at day and nighttime if trained (and tested) using (a) plain color images; (b) just infrared images; and (c) both of them. In order to obtain results for the last item, we propose an early fusion approach to combine features from both modalities. We base the evaluation on a new dataset that we have built for this purpose as well as on the publicly available KAIST multispectral dataset.

[1]  B. Schiele,et al.  Multi-cue onboard pedestrian detection , 2009, 2009 IEEE Conference on Computer Vision and Pattern Recognition.

[2]  Chu-Song Chen,et al.  Fast Human Detection Using a Novel Boosted Cascading Structure With Meta Stages , 2008, IEEE Transactions on Image Processing.

[3]  Pietro Perona,et al.  Integral Channel Features , 2009, BMVC.

[4]  Alexandrina Rogozan,et al.  Pedestrian Detection in Far-Infrared Daytime Images Using a Hierarchical Codebook of SURF , 2015, Sensors.

[5]  Deva Ramanan,et al.  Part-Based Models for Finding People and Estimating Their Pose , 2011, Visual Analysis of Humans.

[6]  Roland Siegwart,et al.  People detection and tracking from aerial thermal views , 2014, 2014 IEEE International Conference on Robotics and Automation (ICRA).

[7]  Charless C. Fowlkes,et al.  Multiresolution Models for Object Detection , 2010, ECCV.

[8]  David Gerónimo Gómez,et al.  2D-3D-based on-board pedestrian detection system , 2010, Comput. Vis. Image Underst..

[9]  Jiaolong Xu,et al.  Domain Adaptation of Deformable Part-Based Models , 2014, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[10]  Corinna Cortes,et al.  Support-Vector Networks , 1995, Machine Learning.

[11]  Antonio M. López,et al.  Adapting Pedestrian Detection from Synthetic to Far Infrared Images , 2013 .

[12]  Shuicheng Yan,et al.  An HOG-LBP human detector with partial occlusion handling , 2009, 2009 IEEE 12th International Conference on Computer Vision.

[13]  Dariu Gavrila,et al.  Monocular Pedestrian Detection: Survey and Experiments , 2009, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[14]  David Gerónimo Gómez,et al.  Survey of Pedestrian Detection for Advanced Driver Assistance Systems , 2010, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[15]  Cristiano Premebida,et al.  Pedestrian detection in far infrared images , 2013, Integr. Comput. Aided Eng..

[16]  Bohyung Han,et al.  Improving object localization using macrofeature layout selection , 2011, 2011 IEEE International Conference on Computer Vision Workshops (ICCV Workshops).

[17]  Paul E. Rybski,et al.  Real-time pedestrian detection with deformable part models , 2012, 2012 IEEE Intelligent Vehicles Symposium.

[18]  Jiaolong Xu,et al.  Multiview random forest of local experts combining RGB and LIDAR data for pedestrian detection , 2015, 2015 IEEE Intelligent Vehicles Symposium (IV).

[19]  David Vázquez,et al.  Random Forests of Local Experts for Pedestrian Detection , 2013, 2013 IEEE International Conference on Computer Vision.

[20]  Paulo Peixoto,et al.  On Exploration of Classifier Ensemble Synergism in Pedestrian Detection , 2010, IEEE Transactions on Intelligent Transportation Systems.

[21]  Dariu Gavrila,et al.  A mixed generative-discriminative framework for pedestrian classification , 2008, 2008 IEEE Conference on Computer Vision and Pattern Recognition.

[22]  Andreas Geiger,et al.  Are we ready for autonomous driving? The KITTI vision benchmark suite , 2012, 2012 IEEE Conference on Computer Vision and Pattern Recognition.

[23]  Luc Van Gool,et al.  Pedestrian detection at 100 frames per second , 2012, 2012 IEEE Conference on Computer Vision and Pattern Recognition.

[24]  Donald Prévost,et al.  Combination of colour and thermal sensors for enhanced object detection , 2007, 2007 10th International Conference on Information Fusion.

[25]  Alexandrina Rogozan,et al.  An Evaluation of the Pedestrian Classification in a Multi-Domain Multi-Modality Setup , 2015, Sensors.

[26]  David A. McAllester,et al.  Object Detection with Discriminatively Trained Part Based Models , 2010, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[27]  Quan Pan,et al.  Active Learning Based Pedestrian Detection in Real Scenes , 2006, 18th International Conference on Pattern Recognition (ICPR'06).

[28]  Sebastian Ramos,et al.  Spatiotemporal Stacked Sequential Learning for Pedestrian Detection , 2014, IbPRIA.

[29]  Guillaume-Alexandre Bilodeau,et al.  An iterative integrated framework for thermal-visible image registration, sensor fusion, and people tracking for video surveillance applications , 2012, Comput. Vis. Image Underst..

[30]  Jürgen Beyerer,et al.  Low Resolution Person Detection with a Moving Thermal Infrared Camera by Hot Spot Classification , 2014, 2014 IEEE Conference on Computer Vision and Pattern Recognition Workshops.

[31]  Xiao Chen,et al.  Multi-spectral pedestrian detection , 2015, Signal Process..

[32]  Namil Kim,et al.  Multispectral pedestrian detection: Benchmark dataset and baseline , 2015, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[33]  Kaiming He,et al.  Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks , 2015, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[34]  A. Shashua,et al.  Pedestrian detection for driving assistance systems: single-frame classification and system level performance , 2004, IEEE Intelligent Vehicles Symposium, 2004.

[35]  Kang Ryoung Park,et al.  Robust Pedestrian Detection by Combining Visible and Thermal Infrared Cameras , 2015, Sensors.

[36]  David Vázquez.  Cool world : domain adaptation of virtual and real worlds for human detection using active learning , 2012 .

[37]  Dariu Gavrila,et al.  A Multilevel Mixture-of-Experts Framework for Pedestrian Classification , 2011, IEEE Transactions on Image Processing.

[38]  Bill Triggs,et al.  Histograms of oriented gradients for human detection , 2005, 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05).

[39]  Pietro Perona,et al.  Pedestrian Detection: An Evaluation of the State of the Art , 2012, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[40]  Xin Li,et al.  Robust Pedestrian Tracking and Recognition from FLIR Video: A Unified Approach via Sparse Coding , 2014, Sensors.

[41]  Bernt Schiele,et al.  New features and insights for pedestrian detection , 2010, 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[42]  Cristiano Premebida,et al.  Pedestrian detection combining RGB and dense LIDAR data , 2014, 2014 IEEE/RSJ International Conference on Intelligent Robots and Systems.