Vision and RFID data fusion for tracking people in crowds by a mobile robot

In this paper, we address the problem of realizing a human following task in a crowded environment. We consider an active perception system, consisting of a camera mounted on a pan-tilt unit and a 360^o RFID detection system, both embedded on a mobile robot. To perform such a task, it is necessary to efficiently track humans in crowds. In a first step, we have dealt with this problem using the particle filtering framework because it enables the fusion of heterogeneous data, which improves the tracking robustness. In a second step, we have considered the problem of controlling the robot motion to make the robot follow the person of interest. To this aim, we have designed a multi-sensor-based control strategy based on the tracker outputs and on the RFID data. Finally, we have implemented the tracker and the control strategy on our robot. The obtained experimental results highlight the relevance of the developed perceptual functions. Possible extensions of this work are discussed at the end of the article.

[1]  John W. McDonough,et al.  A joint particle filter for audio-visual speaker tracking , 2005, ICMI '05.

[2]  Wolfram Burgard,et al.  Using Boosted Features for the Detection of People in 2D Range Data , 2007, Proceedings 2007 IEEE International Conference on Robotics and Automation.

[3]  Hiroshi Ishiguro,et al.  People tracking by cross modal association of vision sensors and acceleration sensors , 2007, 2007 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[4]  Roland Siegwart,et al.  Multimodal People Detection and Tracking in Crowded Scenes , 2008, AAAI.

[5]  Ben J. A. Kröse,et al.  Keeping Track of Humans: Have I Seen This Person Before? , 2005, Proceedings of the 2005 IEEE International Conference on Robotics and Automation.

[6]  Peter Corke,et al.  Visual Control of Robots: High-Performance Visual Servoing , 1996 .

[7]  Tatsuya Nomura,et al.  Analysis of People Trajectories with Ubiquitous Sensors in a Science Museum , 2007, Proceedings 2007 IEEE International Conference on Robotics and Automation.

[8]  Paul A. Viola,et al.  Fast Multi-view Face Detection , 2003 .

[9]  P.K. Dutta,et al.  Audio -Visual Biometric Based Speaker Identification , 2007, International Conference on Computational Intelligence and Multimedia Applications (ICCIMA 2007).

[10]  Michael Isard,et al.  CONDENSATION—Conditional Density Propagation for Visual Tracking , 1998, International Journal of Computer Vision.

[11]  Satoshi Takahashi,et al.  A ZigBee-based sensor node for tracking people's locations , 2008, CASEMANS '08.

[12]  Andrea Cavallaro,et al.  Multi-Modal Particle Filtering Tracking using Appearance, Motion and Audio Likelihoods , 2007, 2007 IEEE International Conference on Image Processing.

[13]  Rachid Alami,et al.  An Architecture for Autonomy , 1998, Int. J. Robotics Res..

[14]  Luca Iocchi,et al.  Person following through appearance models and stereo vision using a mobile robot , 2007, VISAPP.

[15]  Tomomasa Sato,et al.  Multiple people tracking by integrating distributed floor pressure sensors and RFID system , 2004, 2004 IEEE International Conference on Systems, Man and Cybernetics (IEEE Cat. No.04CH37583).

[16]  Patrick Pérez,et al.  Data fusion for visual tracking with particles , 2004, Proceedings of the IEEE.

[17]  Frédéric Lerasle,et al.  Particle filtering strategies for data fusion dedicated to visual tracking from a mobile robot , 2008, Machine Vision and Applications.

[18]  Wolfram Burgard,et al.  Mapping and localization with RFID technology , 2004, IEEE International Conference on Robotics and Automation, 2004. Proceedings. ICRA '04. 2004.

[19]  Roland Siegwart,et al.  Multimodal detection and tracking of pedestrians in urban environments with explicit ground plane extraction , 2008 .

[20]  Rafael Muñoz-Salinas,et al.  Adaptive multi-modal stereo people tracking without background modelling , 2008, J. Vis. Commun. Image Represent..

[21]  Chung-Lin Huang,et al.  Multiple Human Objects Tracking in Crowded Scenes , 2006, 18th International Conference on Pattern Recognition (ICPR'06).

[22]  Ying Li,et al.  Adaptive speaker identification with audiovisual cues for movie content analysis , 2004, Pattern Recognit. Lett..

[23]  Joshua R. Smith,et al.  RFID-based techniques for human-activity detection , 2005, Commun. ACM.

[24]  Frédéric Lerasle,et al.  Video-Based Face Recognition and Tracking from a Robot Companion , 2009, Int. J. Pattern Recognit. Artif. Intell..

[25]  Paul A. Viola,et al.  Rapid object detection using a boosted cascade of simple features , 2001, Proceedings of the 2001 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. CVPR 2001.

[26]  Ryosuke Shibasaki,et al.  Multi-modal tracking of people using laser scanners and video camera , 2008, Image Vis. Comput..

[27]  Illah R. Nourbakhsh,et al.  A survey of socially interactive robots , 2003, Robotics Auton. Syst..

[28]  Neil J. Gordon,et al.  A tutorial on particle filters for online nonlinear/non-Gaussian Bayesian tracking , 2002, IEEE Trans. Signal Process..

[29]  James W. Davis,et al.  Advances in vision algorithms and systems beyond the visible spectrum , 2007, Comput. Vis. Image Underst..

[30]  James L. Crowley,et al.  Localisation intra-bâtiment multi-technologies: RFID, wifi et vision , 2005, UbiMob '05.

[31]  Jungho Lee,et al.  Object tracking method using back-projection of multiple color histogram models , 2003, Proceedings of the 2003 International Symposium on Circuits and Systems, 2003. ISCAS '03..

[32]  Yan Wang,et al.  Real-Time Multi-View Face Detection and Pose Estimation in Video Stream , 2006, 18th International Conference on Pattern Recognition (ICPR'06).

[33]  Huosheng Hu,et al.  Vision and Laser Data Fusion for Tracking People with a Mobile Robot , 2006, 2006 IEEE International Conference on Robotics and Biomimetics.

[34]  Reid G. Simmons,et al.  Natural person-following behavior for social robots , 2007, 2007 2nd ACM/IEEE International Conference on Human-Robot Interaction (HRI).

[35]  Ben J. A. Kröse,et al.  Part based people detection using 2D range data and images , 2007, 2007 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[36]  Yuan Li,et al.  High-Performance Rotation Invariant Multiview Face Detection , 2007, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[37]  Gaurav S. Sukhatme,et al.  People tracking and following with mobile robot using an omnidirectional camera and a laser , 2006, Proceedings 2006 IEEE International Conference on Robotics and Automation, 2006. ICRA 2006..

[38]  N. Gordon,et al.  Novel approach to nonlinear/non-Gaussian Bayesian state estimation , 1993 .

[39]  Patrick Rives,et al.  A new approach to visual servoing in robotics , 1992, IEEE Trans. Robotics Autom..

[40]  Dariu Gavrila,et al.  Multi-cue Pedestrian Detection and Tracking from a Moving Vehicle , 2007, International Journal of Computer Vision.

[41]  Wolfram Burgard,et al.  Tracking multiple moving targets with a mobile robot using particle filters and statistical data association , 2001, Proceedings 2001 ICRA. IEEE International Conference on Robotics and Automation (Cat. No.01CH37164).

[42]  Grzegorz Cielniak,et al.  Improved data association and occlusion handling for vision-based people tracking by mobile robots , 2007, 2007 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[43]  Michael Isard,et al.  ICONDENSATION: Unifying Low-Level and High-Level Tracking in a Stochastic Framework , 1998, ECCV.

[44]  Stefan Roth,et al.  People-tracking-by-detection and people-detection-by-tracking , 2008, 2008 IEEE Conference on Computer Vision and Pattern Recognition.

[45]  Andrei Szabo,et al.  WLAN-Based Pedestrian Tracking Using Particle Filters and Low-Cost MEMS Sensors , 2007, 2007 4th Workshop on Positioning, Navigation and Communication.

[46]  Horst-Michael Groß,et al.  An approach to multi-modal human-machine interaction for intelligent service robots , 2003, Robotics Auton. Syst..

[47]  Dieter Fox,et al.  Knowledge Compilation Properties of Trees-of-BDDs, Revisited , 2009, IJCAI.