Object Detection from Thermal Infrared and Visible Light Cameras in Search and Rescue Scenes

Visual object recognition is a fundamental challenge for reliable search and rescue (SAR) robots, where vision can be limited by lighting and other harsh environmental conditions in disaster sites. The goal of this paper is to explore the use of thermal and visible light images for automatic object detection in SAR scenes. With this purpose, we have used a new dataset consisting of pairs of thermal infrared (TIR) and visible (RGB) video sequences captured from an all-terrain vehicle moving through several realistic SAR exercises participated by actual first response organisations. Two instances of the open source YOLOv3 convolutional neural network (CNN) architecture are trained from annotated sets of RGB and TIR images, respectively. In particular, frames are labelled with four representative classes in SAR scenes comprising both persons (civilian and first-responder) and vehicles (Civilian-car and response-vehicle). Furthermore, we perform a comparative evaluation of these networks that can provide insight for future RGB/TIR fusion.

[1]  P. Rudol,et al.  Human Body Detection and Geolocalization for UAV Search and Rescue Missions Using Color and Thermal Imagery , 2008, 2008 IEEE Aerospace Conference.

[2]  Wei Liu,et al.  SSD: Single Shot MultiBox Detector , 2015, ECCV.

[3]  Pietro Perona,et al.  Microsoft COCO: Common Objects in Context , 2014, ECCV.

[4]  Shu Wang,et al.  Multispectral Deep Neural Networks for Pedestrian Detection , 2016, BMVC.

[5]  Jiayi Ma,et al.  Infrared and visible image fusion methods and applications: A survey , 2018, Inf. Fusion.

[6]  Ayoung Kim,et al.  Sparse Depth Enhanced Direct Thermal-Infrared SLAM Beyond the Visible Spectrum , 2019, IEEE Robotics and Automation Letters.

[7]  In So Kweon,et al.  KAIST Multi-Spectral Day/Night Data Set for Autonomous and Assisted Driving , 2018, IEEE Transactions on Intelligent Transportation Systems.

[8]  Alexander Ferrein,et al.  Evaluation of sensors and mapping approaches for disasters in tunnels , 2013, 2013 IEEE International Symposium on Safety, Security, and Rescue Robotics (SSRR).

[9]  Ronan Fablet,et al.  CNN-Based Target Recognition and Identification for Infrared Imaging in Defense Systems , 2019, Sensors.

[10]  Md Jahidul Islam,et al.  Robotic Detection of Marine Litter Using Deep Visual Detection Models , 2018, 2019 International Conference on Robotics and Automation (ICRA).

[11]  Alois Knoll,et al.  Mixed Frame-/Event-Driven Fast Pedestrian Detection , 2019, 2019 International Conference on Robotics and Automation (ICRA).

[12]  Miran Pobar,et al.  Human Detection in Thermal Imaging Using YOLO , 2019, Proceedings of the 2019 5th International Conference on Computer and Technology Applications.

[13]  Sebastian P. Kleinschmidt,et al.  Visual Multimodal Odometry: Robust Visual Odometry in Harsh Environments , 2018, 2018 IEEE International Symposium on Safety, Security, and Rescue Robotics (SSRR).

[14]  Ali Farhadi,et al.  You Only Look Once: Unified, Real-Time Object Detection , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[15]  Xindong Wu,et al.  Object Detection With Deep Learning: A Review , 2018, IEEE Transactions on Neural Networks and Learning Systems.

[16]  Ludek Zalud,et al.  Fusion of thermal imaging and CCD camera-based data for stereovision visual telepresence , 2013, 2013 IEEE International Symposium on Safety, Security, and Rescue Robotics (SSRR).

[17]  Enrique Alegre,et al.  Una Revisión Sistemática de Métodos para Localizar Automáticamente Objetos en Imágenes , 2018, Revista Iberoamericana de Automática e Informática industrial.

[18]  Guigang Zhang,et al.  Deep Learning , 2016, Int. J. Semantic Comput..

[19]  Yuxiang Sun,et al.  RTFNet: RGB-Thermal Fusion Network for Semantic Segmentation of Urban Scenes , 2019, IEEE Robotics and Automation Letters.

[20]  Seung-Hun Kim,et al.  Thermal Stereo System for Visible Range Extension of Disaster Robot , 2018, 2018 IEEE International Symposium on Safety, Security, and Rescue Robotics (SSRR).

[21]  Serban-Vasile Carata,et al.  Object Recognition on Long Range Thermal Image Using State of the Art DNN , 2018, 2018 Conference Grid, Cloud & High Performance Computing in Science (ROLCG).

[22]  Marilyn Wolf,et al.  CAMEL Dataset for Visual and Thermal Infrared Multiple Object Detection and Tracking , 2018, 2018 15th IEEE International Conference on Advanced Video and Signal Based Surveillance (AVSS).

[23]  Hideo Saito,et al.  Efficient Object-Oriented Semantic Mapping With Object Detector , 2019, IEEE Access.

[24]  Shaowu Peng,et al.  Benchmarking a large-scale FIR dataset for on-road pedestrian detection , 2019, Infrared Physics & Technology.

[25]  Karel Zimmermann,et al.  Simultaneous exploration and segmentation for search and rescue , 2018, J. Field Robotics.

[26]  Ali Farhadi,et al.  YOLOv3: An Incremental Improvement , 2018, ArXiv.

[27]  Taskin Padir,et al.  Autonomous Robot Navigation with Rich Information Mapping in Nuclear Storage Environments , 2018, 2018 IEEE International Symposium on Safety, Security, and Rescue Robotics (SSRR).

[28]  ByoungChul Ko,et al.  Pedestrian Detection at Night Using Deep Neural Networks and Saliency Maps , 2017 .