Self-localizing smart cameras and their applications

As the prices of cameras and computing elements continue to fall, it has become increasingly attractive to consider the deployment of smart camera networks. These networks would be composed of small, networked computers equipped with inexpensive image sensors. Such networks could be employed in a wide range of applications including surveillance, robotics and 3D scene reconstruction. One critical problem that must be addressed before such systems can be deployed effectively is the issue of localization. That is, in order to take full advantage of the images gathered from multiple vantage points it is helpful to know how the cameras in the scene are positioned and oriented with respect to each other. To address the localization problem we have proposed a novel approach to localizing networks of embedded cameras and sensors. In this scheme the cameras and the nodes are equipped with controllable light sources (either visible or infrared) which are used for signaling. Each camera node can then automatically determine the bearing to all the nodes that are visible from its vantage point. By fusing these measurements with the measurements obtained from onboard accelerometers, the camera nodes are able to determine the relative positions and orientations of other nodes in the network. This localization technology can serve as a basic capability on which higher level applications can be built. The method could be used to automatically survey the locations of sensors of interest, to implement distributed surveillance systems or to analyze the structure of a scene based on the images obtained from multiple registered vantage points. It also provides a mechanism for This dissertation is available at ScholarlyCommons: http://repository.upenn.edu/edissertations/579 integrating the imagery obtained from the cameras with the measurements obtained from distributed sensors. We have successfully used our custom made self localizing smart camera networks to implement a novel decentralized target tracking algorithm, create an ad-hoc range finder and localize the components of a self assembling modular robot. Degree Type Dissertation Degree Name Doctor of Philosophy (PhD) Graduate Group Computer and Information Science First Advisor Camillo J. Taylor

[1]  Bernhard Rinner,et al.  Distributed Multilevel Data Fusion for Networked Embedded Systems , 2008, IEEE Journal of Selected Topics in Signal Processing.

[2]  Dariu Gavrila,et al.  Multi-person Tracking with Overlapping Cameras in Complex, Dynamic Environments , 2009, BMVC.

[3]  Mubarak Shah,et al.  A Multiview Approach to Tracking People in Crowded Scenes Using a Planar Homography Constraint , 2006, ECCV.

[4]  Richard Szeliski,et al.  Reconstructing building interiors from images , 2009, 2009 IEEE 12th International Conference on Computer Vision.

[5]  Ramesh Raskar,et al.  Image-based visual hulls , 2000, SIGGRAPH.

[6]  Pier Luigi Dragotti,et al.  Distributed sampling and compression of scenes with finite rate of innovation in camera sensor networks , 2006, Data Compression Conference (DCC'06).

[7]  Prashant J. Shenoy,et al.  SensEye: a multi-tier camera sensor network , 2005, ACM Multimedia.

[8]  Rama Chellappa,et al.  View synthesis of articulating humans using visual hull , 2003, 2003 International Conference on Multimedia and Expo. ICME '03. Proceedings (Cat. No.03TH8698).

[9]  Benjamin Grocholsky,et al.  The robotics bus: a local communications bus for robots , 2004, SPIE Optics East.

[10]  L. Davis,et al.  M2Tracker: A Multi-View Approach to Segmenting and Tracking People in a Cluttered Scene , 2003, International Journal of Computer Vision.

[11]  Manolis I. A. Lourakis,et al.  SBA: A software package for generic sparse bundle adjustment , 2009, TOMS.

[12]  Christiaan J. J. Paredis,et al.  Heterogeneous Teams of Modular Robots for Mapping and Exploration , 2000, Auton. Robots.

[13]  Deborah Estrin,et al.  Cyclops: in situ image sensing and interpretation in wireless sensor networks , 2005, SenSys '05.

[14]  Camillo J. Taylor,et al.  Self-localizing smart camera networks , 2012, TOSN.

[15]  Wu-chi Feng,et al.  Panoptes: scalable low-power video sensor networking technologies , 2003, ACM Multimedia.

[16]  Haeyong Chung,et al.  Distributed metric calibration of ad hoc camera networks , 2006, TOSN.

[17]  Kiriakos N. Kutulakos,et al.  Linear Sequence-to-Sequence Alignment , 2004, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[18]  Brian D. O. Anderson,et al.  On frame and orientation localization for relative sensing networks , 2008, 2008 47th IEEE Conference on Decision and Control.

[19]  Marc Pollefeys,et al.  Camera network calibration from dynamic silhouettes , 2004, Proceedings of the 2004 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2004. CVPR 2004..

[20]  Horst Bischof,et al.  Object Reacquisition and Tracking in Large-Scale Smart Camera Networks , 2007, 2007 First ACM/IEEE International Conference on Distributed Smart Cameras.

[21]  Deborah Estrin,et al.  Exploring Tradeoffs in Accuracy, Energy and Latency of Scale Invariant Feature Transform in Wireless Camera Networks , 2007, 2007 First ACM/IEEE International Conference on Distributed Smart Cameras.

[22]  A. Deshpande,et al.  Mining a World of Smart Sensors , 2002 .

[23]  Gaurav S. Sukhatme,et al.  Connecting the Physical World with Pervasive Networks , 2002, IEEE Pervasive Comput..

[24]  Amit K. Roy-Chowdhury,et al.  Decentralized camera network control using game theory , 2008, 2008 Second ACM/IEEE International Conference on Distributed Smart Cameras.

[25]  David C. Moore,et al.  Robust distributed network localization with noisy range measurements , 2004, SenSys '04.

[26]  C. Guestrin,et al.  Distributed localization of networked cameras , 2006, 2006 5th International Conference on Information Processing in Sensor Networks.

[27]  Andrea Cavallaro,et al.  Global trajectory reconstruction from distributed visual sensors , 2008, 2008 Second ACM/IEEE International Conference on Distributed Smart Cameras.

[28]  H. Aghajan,et al.  A Smart Camera Mote Architecture for Distributed Intelligent Surveillance , 2006 .

[29]  Camillo J. Taylor A scheme for calibrating smart camera networks using active lights , 2004, SenSys '04.

[30]  Janne Heikkilä,et al.  A four-step camera calibration procedure with implicit image correction , 1997, Proceedings of IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[31]  Anthony Cowley,et al.  Distributed sensor databases for multi-robot teams , 2004, IEEE International Conference on Robotics and Automation, 2004. Proceedings. ICRA '04. 2004.

[32]  Rainer Stiefelhagen,et al.  Towards vision-based 3-D people tracking in a smart room , 2002, Proceedings. Fourth IEEE International Conference on Multimodal Interfaces.

[33]  Ryo Kurazume,et al.  Cooperative positioning with multiple robots , 1994, Proceedings of the 1994 IEEE International Conference on Robotics and Automation.

[34]  Mark Yim,et al.  Using Smart Cameras to Localize Self-Assembling Modular Robots , 2007, 2007 First ACM/IEEE International Conference on Distributed Smart Cameras.

[35]  René Vidal,et al.  Distributed image-based 3-D localization of camera sensor networks , 2009, Proceedings of the 48h IEEE Conference on Decision and Control (CDC) held jointly with 2009 28th Chinese Control Conference.

[36]  Gyula Simon,et al.  Sensor network-based countersniper system , 2004, SenSys '04.

[37]  Deborah Estrin,et al.  Self-configuring localization systems: Design and Experimental Evaluation , 2004, TECS.

[38]  Robert Szewczyk,et al.  System architecture directions for networked sensors , 2000, ASPLOS IX.

[39]  Pascal Fua,et al.  Multicamera People Tracking with a Probabilistic Occupancy Map , 2008, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[40]  Bernhard P. Wrobel,et al.  Multiple View Geometry in Computer Vision , 2001 .

[41]  Yael Moses,et al.  Homography based multiple camera detection and tracking of people in a dense crowd , 2008, 2008 IEEE Conference on Computer Vision and Pattern Recognition.

[42]  Lior Wolf,et al.  Sequence-to-Sequence Self Calibration , 2002, ECCV.

[43]  Dinesh,et al.  System Architecture Directions for Networked Sensors , 2014 .

[44]  Henry Medeiros,et al.  Distributed Object Tracking Using a Cluster-Based Kalman Filter in Wireless Camera Networks , 2008, IEEE Journal of Selected Topics in Signal Processing.

[45]  Camillo J. Taylor,et al.  Self Localizing Smart Camera Networks and their Applications to 3 D Modeling , 2006 .

[46]  Zhaolin Cheng,et al.  Calibrating Distributed Camera Networks , 2008, Proceedings of the IEEE.

[47]  Seth J. Teller,et al.  Scalable Extrinsic Calibration of Omni-Directional Image Networks , 2002, International Journal of Computer Vision.

[48]  Steven M. Seitz,et al.  Photo tourism: exploring photo collections in 3D , 2006, ACM Trans. Graph..

[49]  Zhaolin Cheng,et al.  Determining Vision Graphs for Distributed Camera Networks Using Feature Digests , 2007, EURASIP J. Adv. Signal Process..

[50]  Marc Parizeau,et al.  MONNET: Monitoring Pedestrians with a Network of Loosely-Coupled Cameras , 2006, 18th International Conference on Pattern Recognition (ICPR'06).

[51]  Richard Szeliski,et al.  Building Rome in a day , 2009, 2009 IEEE 12th International Conference on Computer Vision.

[52]  Ryo Kurazume,et al.  Study on cooperative positioning system (basic principle and measurement experiment) , 1996, Proceedings of IEEE International Conference on Robotics and Automation.

[53]  Marc Pollefeys,et al.  Pan-tilt-zoom camera calibration and high-resolution mosaic generation , 2006, Comput. Vis. Image Underst..

[54]  Denis Simakov,et al.  Feature-Based Sequence-to-Sequence Matching , 2006, International Journal of Computer Vision.

[55]  Trevor Darrell,et al.  Simultaneous calibration and tracking with a network of non-overlapping sensors , 2004, Proceedings of the 2004 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2004. CVPR 2004..

[56]  Tinne Tuytelaars,et al.  Synchronizing video sequences , 2004, CVPR 2004.

[57]  Dimitrios Lymberopoulos,et al.  Sensor Localization and Camera Calibration in Distributed Camera Sensor Networks , 2006, 2006 3rd International Conference on Broadband Communications, Networks and Systems.

[58]  Gregory Dudek,et al.  Multi-robot cooperative localization: a study of trade-offs between efficiency and accuracy , 2002, IEEE/RSJ International Conference on Intelligent Robots and Systems.

[59]  Ryo Kurazume,et al.  Study on cooperative positioning system: optimum moving strategies for CPS-III , 1998, Proceedings. 1998 IEEE International Conference on Robotics and Automation (Cat. No.98CH36146).

[60]  Wolfgang Straßer,et al.  3D Surveillance A Distributed Network of Smart Cameras for Real-Time Tracking and its Visualization in 3D , 2006, 2006 Conference on Computer Vision and Pattern Recognition Workshop (CVPRW'06).

[61]  Srinivasan Seshan,et al.  IrisNet: An Architecture for Compute-Intensive Wide-Area Sensor Network Services , 2002 .

[62]  Björn W. Schuller,et al.  Applying multi layer homography for multi camera person tracking , 2008, 2008 Second ACM/IEEE International Conference on Distributed Smart Cameras.

[63]  John J. Leonard,et al.  Pure range-only sub-sea SLAM , 2003, 2003 IEEE International Conference on Robotics and Automation (Cat. No.03CH37422).

[64]  B. S. Manjunath,et al.  VISNET: A distributed vision testbed , 2008, 2008 Second ACM/IEEE International Conference on Distributed Smart Cameras.

[65]  Robert H. Halstead,et al.  Matrix Computations , 2011, Encyclopedia of Parallel Computing.

[66]  Wayne H. Wolf,et al.  A peer-to-peer architecture for distributed real-time gesture recognition , 2004, 2004 IEEE International Conference on Multimedia and Expo (ICME) (IEEE Cat. No.04TH8763).

[67]  Ying Zhang,et al.  STAM: a system of tracking and mapping in real environments , 2004, IEEE Wireless Communications.

[68]  Mark Yim,et al.  Automatic Configuration Recognition Methods in Modular Robots , 2008, Int. J. Robotics Res..