Real-time 3D human tracking for mobile robots with multisensors

Acquiring the accurate 3-D position of a target person around a robot provides fundamental and valuable information that is applicable to a wide range of robotic tasks, including home service, navigation and entertainment. This paper presents a real-time robotic 3-D human tracking system which combines a monocular camera with an ultrasonic sensor by the extended Kalman filter (EKF). The proposed system consists of three sub-modules: monocular camera sensor tracking model, ultrasonic sensor tracking model and multi-sensor fusion. An improved visual tracking algorithm is presented to provide partial location estimation (2-D). The algorithm is designed to overcome severe occlusions, scale variation, target missing and achieve robust re-detection. The scale accuracy is further enhanced by the estimated 3-D information. An ultrasonic sensor array is employed to provide the range information from the target person to the robot and Gaussian Process Regression is used for partial location estimation (2-D). EKF is adopted to sequentially process multiple, heterogeneous measurements arriving in an asynchronous order from the vision sensor and the ultrasonic sensor separately. In the experiments, the proposed tracking system is tested in both simulation platform and actual mobile robot for various indoor and outdoor scenes. The experimental results show the superior performance of the 3-D tracking system in terms of both the accuracy and robustness.

[1]  Zdenek Kalal,et al.  Tracking-Learning-Detection , 2012, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[2]  Seoyong Shin,et al.  Integrated collision avoidance and tracking system for mobile robot , 2012, 2012 International Conference of Robotics and Artificial Intelligence.

[3]  Ryosuke Shibasaki,et al.  Multi-modal tracking of people using laser scanners and video camera , 2008, Image Vis. Comput..

[4]  Pramod K. Varshney,et al.  Multisensor Data Fusion , 1997, IEA/AIE.

[5]  Marc Hanheide,et al.  Real-time multisensor people tracking for human-robot spatial interaction , 2015 .

[6]  Michael Felsberg,et al.  The Visual Object Tracking VOT2015 Challenge Results , 2015, 2015 IEEE International Conference on Computer Vision Workshop (ICCVW).

[7]  K.V. Kumar,et al.  Ultra Sonic Sensor Based Blind Spot Accident Prevention System , 2008, 2008 International Conference on Advanced Computer Theory and Engineering.

[8]  Huosheng Hu,et al.  Multisensor-Based Human Detection and Tracking for Mobile Service Robots , 2009, IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics).

[9]  Gang Wang,et al.  Real-time part-based visual tracking via adaptive correlation filters , 2015, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[10]  Frédéric Lerasle,et al.  Vision and RFID data fusion for tracking people in crowds by a mobile robot , 2010, Comput. Vis. Image Underst..

[11]  Luca Maria Gambardella,et al.  Kinect-based people detection and tracking from small-footprint ground robots , 2014, 2014 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[12]  Bill Triggs,et al.  Histograms of oriented gradients for human detection , 2005, 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05).

[13]  Rüdiger Dillmann,et al.  Sensor fusion for 3D human body tracking with an articulated 3D body model , 2006, Proceedings 2006 IEEE International Conference on Robotics and Automation, 2006. ICRA 2006..

[14]  Rui Caseiro,et al.  Ieee Transactions on Pattern Analysis and Machine Intelligence High-speed Tracking with Kernelized Correlation Filters , 2022 .

[15]  Ahmad B. Rad,et al.  Heterogeneous multisensor fusion for mapping dynamic environments , 2007, Adv. Robotics.

[16]  Michael Felsberg,et al.  Accurate Scale Estimation for Robust Visual Tracking , 2014, BMVC.

[17]  Matteo Munaro,et al.  Fast RGB-D people tracking for service robots , 2014, Auton. Robots.

[18]  Ming-Hsuan Yang,et al.  Hierarchical Convolutional Features for Visual Tracking , 2015, 2015 IEEE International Conference on Computer Vision (ICCV).

[19]  Mengmeng Wang,et al.  Robust object tracking with a hierarchical ensemble framework , 2015, 2016 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS).

[20]  Horst-Michael Gross,et al.  Sensor Fusion for Vision and Sonar Based People Tracking on a Mobile Service Robot , 2002 .

[21]  Changsheng Xu,et al.  Structural Correlation Filter for Robust Visual Tracking , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[22]  Gaurav S. Sukhatme,et al.  People tracking and following with mobile robot using an omnidirectional camera and a laser , 2006, Proceedings 2006 IEEE International Conference on Robotics and Automation, 2006. ICRA 2006..

[23]  Michael Felsberg,et al.  Adaptive Color Attributes for Real-Time Visual Tracking , 2014, 2014 IEEE Conference on Computer Vision and Pattern Recognition.

[24]  Jaime Valls Miró,et al.  An ultrasonic/RF GP-based sensor model robotic solution for indoors/outdoors person tracking , 2014, 2014 13th International Conference on Control Automation Robotics & Vision (ICARCV).

[25]  Ming-Hsuan Yang,et al.  Object Tracking Benchmark , 2015, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[26]  Jong-Hann Jean,et al.  Development of an indoor patrol robot based on ultrasonic and vision data fusion , 2013, 2013 IEEE International Conference on Mechatronics and Automation.