A system for real-time panorama generation and display in tele-immersive applications

Wide field-of-view (FOV) is necessary for many industrial applications, such as air traffic control, large vehicle driving and navigation. Unfortunately, the supporting structure/frame in most systems usually blocks part of the view, results in "blind spot" and raises the risk to the pilot. In this paper, we introduce a video-based tele-immersive system, called the immersive cockpit. It captures live videos from the working site and recreates an immersive environment at the remote site where the pilot situates. It immerses the pilot at the remote site with a panoramic view of the environment, and hence improves interactivity and safety. The design goals of our system are real-time, live, low-cost, and scalable. We stitch multiple video streams captured from ordinary charged couple device cameras to generate a panoramic video. To avoid being blocked by the supporting frame, we allow a flexible placement of cameras. This approach trades the accuracy of the generated panoramic image for a larger FOV. To reduce the computation, parameters for stitching are determined once during the system initialization. The panoramic video is presented on an immersive display which covers the FOV of the viewer. We discuss how to correctly present the panoramic video on this nonplanar immersive display screen by sweet spot relocation. We also present the result and the performance evaluation of the system.

[1]  Prashant J. Shenoy,et al.  Issues in multimedia server design , 1995, CSUR.

[2]  Carolina Cruz-Neira,et al.  Surround-Screen Projection-Based Virtual Reality: The Design and Implementation of the CAVE , 2023 .

[3]  W. Brent Seales,et al.  Immersive teleconferencing: a new algorithm to generate seamless panoramic video imagery , 1999, MULTIMEDIA '99.

[4]  Michitaka Hirose,et al.  Integrating Live Video for Immersive Environments , 1999, IEEE Multim..

[5]  Domenico Ferrari,et al.  Design and Applications of a Delay Jitter Control Scheme for Packet-Switching Internetworks , 1991, NOSSDAV.

[6]  Kevin Jeffay,et al.  Kernel Support for Live Digital Audio and Video , 1991, NOSSDAV.

[7]  James Davis,et al.  Mosaics of scenes with moving objects , 1998, Proceedings. 1998 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (Cat. No.98CB36231).

[8]  Craig Partridge.  Isochronous applications do not require jitter-controlled networks , 1991, RFC.

[9]  Henning Schulzrinne,et al.  Real Time Streaming Protocol (RTSP) , 1998, RFC.

[10]  C. D. Kuglin,et al.  The phase correlation image alignment method , 1975 .

[11]  Olivier Faugeras,et al.  Three-Dimensional Computer Vision , 1993 .

[12]  Edward H. Adelson,et al.  A multiresolution spline with application to image mosaics , 1983, TOGS.

[13]  Harpreet S. Sawhney,et al.  Robust Video Mosaicing through Topology Inference and Local to Global Alignment , 1998, ECCV.

[14]  Kenneth Turkowski,et al.  Creating image-based VR using a self-calibrating fisheye lens , 1997, Proceedings of IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[15]  Greg Welch,et al.  The office of the future: a unified approach to image-based modeling and spatially immersive displays , 1998, SIGGRAPH.

[16]  Jean Rouch,et al.  The Cinema of the Future , 1985 .

[17]  S. Nayar,et al.  Polycameras: Camera Clusters for Wide Angle Imaging , 1999 .

[18]  Morton Leonard Heilig,et al.  EL Cine del Futuro: The Cinema of the Future , 1992, Presence: Teleoperators & Virtual Environments.

[19]  Kyoung Shin Park,et al.  Effects of network characteristics on human performance in a collaborative virtual environment , 1999, Proceedings IEEE Virtual Reality (Cat. No. 99CB36316).

[20]  Richard Szeliski,et al.  Creating full view panoramic image mosaics and texture-mapped models , 1997, International Conference on Computer Graphics and Interactive Techniques.

[21]  Roger Y. Tsai,et al.  A versatile camera calibration technique for high-accuracy 3D machine vision metrology using off-the-shelf TV cameras and lenses , 1987, IEEE J. Robotics Autom..

[22]  Ramesh Govindan,et al.  Support for continuous media in the DASH system , 1990, Proceedings.,10th International Conference on Distributed Computing Systems.

[23]  David L. Milgram,et al.  Adaptive Techniques for Photomosaicking , 1977, IEEE Transactions on Computers.

[24]  Martin R. Varley,et al.  Improved Video Mosaic Construction by Accumulated Alignment Error Distribution , 1998, BMVC.

[25]  Randall Frank,et al.  High-Resolution Multiprojector Display Walls , 2000, IEEE Computer Graphics and Applications.

[26]  Shmuel Peleg,et al.  Panoramic mosaics by manifold projection , 1997, Proceedings of IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[27]  Richard Szeliski,et al.  Image mosaicing for tele-reality applications , 1994, Proceedings of 1994 IEEE Workshop on Applications of Computer Vision.

[28]  W. Brent Seales,et al.  Multi-projector displays using camera-based registration , 1999, Proceedings Visualization '99 (Cat. No.99CB37067).

[29]  Subhasis Chaudhuri,et al.  Automated assembling of images: image montage preparation , 1995, Pattern Recognit..

[30]  Anup Basu,et al.  Panoramic video with predictive windows for telepresence applications , 1999, Proceedings 1999 IEEE International Conference on Robotics and Automation (Cat. No.99CH36288C).

[31]  Heung-Yeung Shum,et al.  Panoramic Image Mosaics , 1998 .

[32]  Harry Shum,et al.  Rendering with concentric mosaics , 1999, SIGGRAPH.

[33]  Richard Szeliski,et al.  Creating full view panoramic image mosaics and environment maps , 1997, SIGGRAPH.

[34]  Shree K. Nayar,et al.  Real-Time Omnidirectional and Panoramic Stereo , 1998 .

[35]  Kostas Daniilidis,et al.  View-independent scene acquisition for tele-presence , 2000, Proceedings IEEE and ACM International Symposium on Augmented Reality (ISAR 2000).

[36]  Ramesh Raskar,et al.  Immersive planar display using roughly aligned projectors , 2000, Proceedings IEEE Virtual Reality 2000 (Cat. No.00CB37048).

[37]  B. N. Chatterji,et al.  An FFT-based technique for translation, rotation, and scale-invariant image registration , 1996, IEEE Trans. Image Process..

[38]  Richard W. Hall,et al.  Mosaic image generation on a flattened Gaussian sphere , 1996, Proceedings Third IEEE Workshop on Applications of Computer Vision. WACV'96.

[39]  Janne Heikkilä,et al.  A four-step camera calibration procedure with implicit image correction , 1997, Proceedings of IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[40]  Shree K. Nayar,et al.  Generation of Perspective and Panoramic Video from Omnidirectional Video , 1997 .

[41]  S. Nayar.  Omnidirectional Video Camera , 1997 .

[42]  Don Kimber,et al.  FlyCam: practical panoramic video and automatic camera control , 2000, 2000 IEEE International Conference on Multimedia and Expo. ICME2000. Proceedings. Latest Advances in the Fast Changing World of Multimedia (Cat. No.00TH8532).

[43]  Richard Szeliski,et al.  Video mosaics for virtual environments , 1996, IEEE Computer Graphics and Applications.