RGB-D, Laser and Thermal Sensor Fusion for People following in a Mobile Robot

Detecting and tracking people is a key capability for robots that operate in populated environments. In this paper, we used a multiple sensor fusion approach that combines three kinds of sensors in order to detect people using RGB-D vision, lasers and a thermal sensor mounted on a mobile platform. The Kinect sensor offers a rich data set at a significantly low cost, however, there are some limitations to its use in a mobile platform, mainly that the Kinect algorithms for people detection rely on images captured by a static camera. To cope with these limitations, this work is based on the combination of the Kinect and a Hokuyo laser and a thermopile array sensor. A real-time particle filter system merges the information provided by the sensors and calculates the position of the target, using probabilistic leg and thermal patterns, image features and optical flow to this end. Experimental results carried out with a mobile platform in a Science museum have shown that the combination of different sensory cues increases the reliability of the people following system.

[1]  Toby Sharp,et al.  Real-time human pose recognition in parts from single depth images , 2011, CVPR.

[2]  Bernt Schiele,et al.  Visual People Detection - Different Models, Comparison and Discussion , 2009, ICRA 2009.

[3]  Mohan M. Trivedi,et al.  Head Pose Estimation in Computer Vision: A Survey , 2009, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[4]  Grzegorz Cielniak,et al.  People tracking by mobile robots using thermal and colour vision , 2007 .

[5]  Ryo Kurazume,et al.  Multi-Part People Detection Using 2D Range Data , 2010, Int. J. Soc. Robotics.

[6]  Andrew W. Fitzgibbon,et al.  Real-time human pose recognition in parts from single depth images , 2011, CVPR 2011.

[7]  Loreto Susperregi,et al.  Laser Based People Following Behaviour in an Emergency Environment , 2009, ICIRA.

[8]  Narendra Ahuja,et al.  Detecting Faces in Images: A Survey , 2002, IEEE Trans. Pattern Anal. Mach. Intell..

[9]  Peter Bajcsy,et al.  Integration of thermal and visible imagery for robust foreground detection in tele-immersive spaces , 2008, 2008 11th International Conference on Information Fusion.

[10]  Neil J. Gordon,et al.  A tutorial on particle filters for online nonlinear/non-Gaussian Bayesian tracking , 2002, IEEE Trans. Signal Process..

[11]  B. Schiele,et al.  Fast and Robust Face Finding via Local Context , 2003 .

[12]  Huosheng Hu,et al.  Multisensor data fusion for joint people tracking and identification with a service robot , 2007, 2007 IEEE International Conference on Robotics and Biomimetics (ROBIO).

[13]  Vijayan K. Asari,et al.  Face recognition in multi-sensor images based on a novel modular feature selection technique , 2010, Inf. Fusion.

[14]  Shuzhi Sam Ge,et al.  Robust Human Detection and Identification by Using Stereo and thermal Images in Human Robot Interaction , 2007, Int. J. Inf. Acquis..

[15]  Sander Oude Elberink,et al.  Accuracy and Resolution of Kinect Depth Data for Indoor Mapping Applications , 2012, Sensors.

[16]  Takeo Kanade,et al.  An Iterative Image Registration Technique with an Application to Stereo Vision , 1981, IJCAI.

[17]  Huosheng Hu,et al.  A Bank of Unscented Kalman Filters for Multimodal Human Perception with Mobile Service Robots , 2010, Int. J. Soc. Robotics.

[18]  Morgan Quigley,et al.  ROS: an open-source Robot Operating System , 2009, ICRA 2009.

[19]  Bill Triggs,et al.  Histograms of oriented gradients for human detection , 2005, 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05).

[20]  Martin Hofmann,et al.  Fusion of Multi-Modal Sensors in a Voxel Occupancy Grid for Tracking and Behaviour Analysis , 2011, WIAMIS 2011.

[21]  Hideya Takahashi,et al.  Fusion of Infrared and Visible Images for Robust Person Detection , 2011 .

[22]  Tamaki Ura,et al.  Sonar and vision based navigation schemes for autonomous underwater vehicles , 2004, ICARCV 2004 8th Control, Automation, Robotics and Vision Conference, 2004..

[23]  Tomaso A. Poggio,et al.  A Trainable System for Object Detection , 2000, International Journal of Computer Vision.

[24]  Grzegorz Cielniak,et al.  Active people recognition using thermal and grey images on a mobile security robot , 2005, 2005 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[25]  Erik Hjelmås,et al.  Face Detection: A Survey , 2001, Comput. Vis. Image Underst..