Target tracking in heterogeneous sensor networks using audio and video sensor fusion

Heterogeneous sensor networks (HSNs) with multiple sensing modalities are gaining popularity in diverse fields. Tracking is an application that can benefit from multiple sensing modalities. If a moving target emits sound then both audio and video sensors can be utilized. These modalities can complement each other in the presence of high background noise that impairs the audio or visual clutter affecting the video. Audio-video tracking can also provide cues for the other modality for actuation. In this paper, we describe an approach for target tracking in urban environments utilizing an HSN of mote class devices equipped with acoustic sensor boards and embedded PCs equipped with web cameras. Our system employs a Markov Chain Monte Carlo Data Association algorithm for tracking vehicles emitting engine noise. Experimental results from a deployment in an urban environment are used to demonstrate our approach.

[1]  C.-C. Jay Kuo,et al.  Automatic Real-Time Moving Target Detection from Infrared Video , 2006, 2006 International Conference on Intelligent Information Hiding and Multimedia.

[2]  Vincent H. Berk,et al.  An overview of process query systems , 2004, SPIE Defense + Commercial Sensing.

[3]  P. KaewTrakulPong,et al.  An Improved Adaptive Background Mixture Model for Real-time Tracking with Shadow Detection , 2002 .

[4]  Stuart J. Russell,et al.  Image Segmentation in Video Sequences: A Probabilistic Approach , 1997, UAI.

[5]  Sascha Spors,et al.  Joint audio-video object localization and tracking , 2001 .

[6]  Y. Bar-Shalom.  Tracking and data association , 1988 .

[7]  Kentaro Toyama,et al.  Wallflower: principles and practice of background maintenance , 1999, Proceedings of the Seventh IEEE International Conference on Computer Vision.

[8]  James J. Little,et al.  A Boosted Particle Filter: Multitarget Detection and Tracking , 2004, ECCV.

[9]  Ramakant Nevatia,et al.  Tracking multiple humans in crowded environment , 2004, Proceedings of the 2004 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2004. CVPR 2004..

[10]  David E. Culler,et al.  Elapsed time on arrival: a simple and versatile primitive for canonical time synchronisation services , 2006, Int. J. Ad Hoc Ubiquitous Comput..

[11]  Aubrey B. Poore,et al.  Multidimensional Assignment Problems Arising in Multitarget and Multisensor Tracking , 2000 .

[12]  Frank Dellaert,et al.  MCMC-based particle filtering for tracking a variable number of interacting targets , 2005, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[13]  Trevor Darrell,et al.  A Probabilistic Framework for Multi-modal Multi-Person Tracking , 2003, 2003 Conference on Computer Vision and Pattern Recognition Workshop.

[14]  Deborah Estrin,et al.  Coherent acoustic array processing and localization on wireless sensor networks , 2003, Proc. IEEE.

[15]  Kung Yao,et al.  Acoustic Source Localization and Beamforming: Theory and Practice , 2003, EURASIP J. Adv. Signal Process..

[16]  Volkan Cevher,et al.  Target Tracking Using a Joint Acoustic Video System , 2007, IEEE Transactions on Multimedia.

[17]  S. Shankar Sastry,et al.  Tracking and Coordination of Multiple Agents Using Sensor Networks: System Design, Algorithms and Experiments , 2007, Proceedings of the IEEE.

[18]  Ingemar J. Cox,et al.  A review of statistical data association techniques for motion correspondence , 1993, International Journal of Computer Vision.

[19]  Suresh Singh,et al.  Exploiting heterogeneity in sensor networks , 2005, Proceedings IEEE 24th Annual Joint Conference of the IEEE Computer and Communications Societies..

[20]  Saurabh Ganeriwal,et al.  Timing-sync protocol for sensor networks , 2003, SenSys '03.

[21]  J. Elson,et al.  Fine-grained network time synchronization using reference broadcasts , 2002, OSDI '02.

[22]  W. Eric L. Grimson,et al.  Learning Patterns of Activity Using Real-Time Tracking , 2000, IEEE Trans. Pattern Anal. Mach. Intell..

[23]  Songhwai Oh,et al.  Markov chain Monte Carlo data association for general multiple-target tracking problems , 2004, 2004 43rd IEEE Conference on Decision and Control (CDC) (IEEE Cat. No.04CH37601).

[24]  Gyula Simon,et al.  Countersniper system for urban warfare , 2005, TOSN.

[25]  Feng Zhao,et al.  Collaborative In-Network Processing for Target Tracking , 2003, EURASIP J. Adv. Signal Process..