Real-Time Landing Spot Detection and Pose Estimation on Thermal Images Using Convolutional Neural Networks

This paper presents a robust, accurate and real-time approach to detect landing spot position and orientation information using deep convolutional neural networks and image processing technique on thermal images. The proposed novel algorithm pipeline consists of two steps: ledge detection and orientation information extraction. The extracted pose information of the landing spot from thermal images could be used to facilitate autonomous operations of unmanned aerial vehicles (UAVs) in both of day and night time. In order to land on the narrow and long ledge, UAV requires accurate orientation information of the ledge. Moreover, the method is scale and rotation invariant and also robust to occlusion in certain special and unexpected situations. Our algorithm runs at 20 frames per second on NVIDIA GTX 1080Ti GPU with the real flight thermal image dataset captured by T-Lion UAV developed by Temasek Laboratories@NUS.

[1]  Trevor Darrell,et al.  Fully Convolutional Networks for Semantic Segmentation , 2017, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[2]  Bill Triggs,et al.  Histograms of oriented gradients for human detection , 2005, 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05).

[3]  Geoffrey E. Hinton,et al.  ImageNet classification with deep convolutional neural networks , 2012, Commun. ACM.

[4]  Zheng Liu,et al.  Pedestrian detection from thermal images: A sparse representation based approach , 2016 .

[5]  G LoweDavid,et al.  Distinctive Image Features from Scale-Invariant Keypoints , 2004 .

[6]  Trevor Darrell,et al.  Rich Feature Hierarchies for Accurate Object Detection and Semantic Segmentation , 2013, 2014 IEEE Conference on Computer Vision and Pattern Recognition.

[7]  Rafael Grompone von Gioi,et al.  LSD: A Fast Line Segment Detector with a False Detection Control , 2010, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[8]  Richard O. Duda,et al.  Use of the Hough transformation to detect lines and curves in pictures , 1972, CACM.

[9]  Feng Lin,et al.  Patch-based keypoints consensus voting for robust visual tracking , 2016, IECON 2016 - 42nd Annual Conference of the IEEE Industrial Electronics Society.

[10]  Li Fei-Fei,et al.  ImageNet: A large-scale hierarchical image database , 2009, CVPR.

[11]  Kaiming He,et al.  Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks , 2015, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[12]  David A. McAllester,et al.  Object Detection with Discriminatively Trained Part Based Models , 2010, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[13]  Ben M. Chen,et al.  System integration of a vision-guided UAV for autonomous landing on moving platform , 2016, 2016 12th IEEE International Conference on Control and Automation (ICCA).

[14]  Ali Farhadi,et al.  You Only Look Once: Unified, Real-Time Object Detection , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[15]  Feng Lin,et al.  Structural keypoints voting for global visual tracking , 2016, 2016 IEEE International Conference on Robotics and Biomimetics (ROBIO).

[16]  Ross B. Girshick,et al.  Fast R-CNN , 2015, 1504.08083.

[17]  Ali Farhadi,et al.  YOLO9000: Better, Faster, Stronger , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[18]  Daewon Lee,et al.  Autonomous landing of a VTOL UAV on a moving platform using image-based visual servoing , 2012, 2012 IEEE International Conference on Robotics and Automation.

[19]  Robert E. Schapire,et al.  A Brief Introduction to Boosting , 1999, IJCAI.

[20]  Paul A. Viola,et al.  Rapid object detection using a boosted cascade of simple features , 2001, Proceedings of the 2001 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. CVPR 2001.

[21]  Eunhyeok Park,et al.  Compression of Deep Convolutional Neural Networks for Fast and Low Power Mobile Applications , 2015, ICLR.

[22]  Koen E. A. van de Sande,et al.  Selective Search for Object Recognition , 2013, International Journal of Computer Vision.

[23]  Ben M. Chen,et al.  Vision-aided tracking of a moving ground vehicle with a hybrid UAV , 2017, 2017 13th IEEE International Conference on Control & Automation (ICCA).