Aligning windows of live video from an imprecise pan-tilt-zoom camera into a remote panoramic display for remote nature observation

A pan-tilt-zoom (PTZ) robotic camera can provide a detailed live video of selected areas of interest within a large potential viewing field. The selective coverage is ideal for nature observation applications where power and bandwidth are often limited. To provide the spatial context for human observers, it is desirable to insert the live video into a large spherical panoramic display representing the entire viewing field. Accurate alignment of the video stream within the panoramic display is difficult due to imprecise pan-tilt values and rapid changes in camera configurations. Common image alignment algorithms are computationally expensive for real time applications. We are interested in designing algorithms that fit low power computation platform and hence can be implemented inside the PTZ camera in the future. We present a sampling-based constant-time image alignment algorithm based on spherical projection and projection-invariant selective sampling that accurately registers paired images at 100 frames per second on a simulated embedded platform. The alignment accuracy actually is better than existing methods when high rotational difference is involved. Experiments suggest that the new alignment algorithm is faster than existing algorithms by 1,471.6 times when aligning a six-mega-pixel image pair.

[1]  Shree K. Nayar,et al.  Catadioptric omnidirectional camera , 1997, Proceedings of IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[2]  Emanuele Trucco,et al.  Dynamic Video Mosaics and Augmented Reality for Subsea Inspection and Monitoring , 2000 .

[3]  Bo Hu,et al.  Acquiring an Environment Map through Image Mosaicking , 2001 .

[4]  Seth J. Teller,et al.  Spherical Mosaics with Quaternions and Dense Correlation , 2000, International Journal of Computer Vision.

[5]  Don Kimber,et al.  Enhancing distance learning with panoramic video , 2001, Proceedings of the 34th Annual Hawaii International Conference on System Sciences.

[6]  Kenneth Turkowski,et al.  Creating image-based VR using a self-calibrating fisheye lens , 1997, Proceedings of IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[7]  Kim Bo Youn,et al.  Adaptive Strip Compression for Panorama Video Streaming , 2006 .

[8]  Dezhen Song,et al.  Systems, control models, and codec for collaborative observation of remote environments with an autonomous networked robotic camera , 2008, Auton. Robots.

[9]  Narendra Ahuja,et al.  Multiview panoramic cameras using mirror pyramids , 2004, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[10]  Min-Cheol Hong,et al.  Low-Hardware-Cost Motion Estimation with Large Search Range for VLSI Multimedia Processors , 2005, IEICE Trans. Inf. Syst..

[11]  Dezhen Song,et al.  Approximate Algorithms for a Collaboratively Controlled Robotic Camera , 2007, IEEE Transactions on Robotics.

[12]  Bernhard P. Wrobel,et al.  Multiple View Geometry in Computer Vision , 2001 .

[13]  Richard I. Hartley,et al.  Self-Calibration of Stationary Cameras , 1997, International Journal of Computer Vision.

[14]  Rafael C. González,et al.  Local Determination of a Moving Contrast Edge , 1985, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[15]  Gregory Doumenis,et al.  A real-time motion estimation FPGA architecture , 2007, Journal of Real-Time Image Processing.

[16]  Rachid Deriche,et al.  Using geometric corners to build a 2D mosaic from a set of images , 1997, Proceedings of IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[17]  Luc Van Gool,et al.  SURF: Speeded Up Robust Features , 2006, ECCV.

[18]  Richard Szeliski,et al.  Efficiently registering video into panoramic mosaics , 2005, Tenth IEEE International Conference on Computer Vision (ICCV'05) Volume 1.

[19]  Shree K. Nayar,et al.  A Theory of Single-Viewpoint Catadioptric Image Formation , 1999, International Journal of Computer Vision.

[20]  Ian T. Jolliffe,et al.  Principal Component Analysis , 2002, International Encyclopedia of Statistical Science.

[21]  Jaeyeon Lee,et al.  Automatic Image Mosaic System Using Image Feature Detection and Taylor Series , 2003, DICTA.

[22]  Christopher G. Harris,et al.  A Combined Corner and Edge Detector , 1988, Alvey Vision Conference.

[23]  B. N. Chatterji,et al.  An FFT-based technique for translation, rotation, and scale-invariant image registration , 1996, IEEE Trans. Image Process..

[24]  Andrew Zisserman,et al.  Feature Based Methods for Structure and Motion Estimation , 1999, Workshop on Vision Algorithms.

[25]  Wei Zhang,et al.  Mosaic construction from a sparse set of views , 2002, Proceedings. First International Symposium on 3D Data Processing Visualization and Transmission.

[26]  H. Ishiguro,et al.  Panoramic Vision , 2001, Monographs in Computer Science.

[27]  C. Morandi,et al.  Registration of Translated and Rotated Images Using Finite Fourier Transforms , 1987, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[28]  Richard Szeliski,et al.  Image Alignment and Stitching , 2006, Handbook of Mathematical Models in Computer Vision.

[29]  Don Kimber,et al.  FlyCam: practical panoramic video and automatic camera control , 2000, 2000 IEEE International Conference on Multimedia and Expo. ICME2000. Proceedings. Latest Advances in the Fast Changing World of Multimedia (Cat. No.00TH8532).

[30]  Andrew W. Fitzgibbon,et al.  Bundle Adjustment - A Modern Synthesis , 1999, Workshop on Vision Algorithms.

[31]  Dezhen Song,et al.  A minimum variance calibration algorithm for pan-tilt robotic cameras in natural environments , 2006, Proceedings 2006 IEEE International Conference on Robotics and Automation, 2006. ICRA 2006..

[32]  Kenichi Kanatani,et al.  Image mosaicing by stratified matching , 2004, Image Vis. Comput..

[33]  Jean-Luc Dugelay,et al.  Using GPU for fast Block-Matching , 2006, 2006 14th European Signal Processing Conference.

[34]  Rachid Deriche,et al.  A Robust Technique for Matching two Uncalibrated Images Through the Recovery of the Unknown Epipolar Geometry , 1995, Artif. Intell..

[35]  Heung-Yeung Shum,et al.  Panoramic Image Mosaics , 1998 .

[36]  Matthew A. Brown,et al.  Recognising panoramas , 2003, Proceedings Ninth IEEE International Conference on Computer Vision.

[37]  David Salesin,et al.  Panoramic video textures , 2005, ACM Trans. Graph..

[38]  Dezhen Song.  Sharing a Vision - Systems and Algorithms for Collaboratively-Teleoperated Robotic Cameras , 2009, Springer Tracts in Advanced Robotics.

[39]  Dana H. Ballard,et al.  Computer Vision , 1982 .