Diminished reality using plane-sweep algorithm with weakly-calibrated cameras

We present a plane-sweep algorithm for removing occluding objects in front of the objective scene from multiple weakly-calibrated cameras. Projective grid space (PGS), a weak cameras calibration framework, is used to obtain geometrical relations between cameras. Plane-sweep algorithm works by implicitly reconstructing the depth maps of the targeted view. By excluding the occluding objects from the volume of the sweeping planes, we can generate new views without the occluding objects. The results show the effectiveness of the proposed method and it is fast enough to run in several frames per second on a consumer PC by implementing the proposed plane-sweep algorithm in graphics processing unit (GPU).

[1]  Edward H. Adelson,et al.  Representing moving images with layers , 1994, IEEE Trans. Image Process..

[2]  Geys,et al.  The augmented auditorium: fast interpolated and augmented view generation , 2006 .

[3]  Hideo Saito,et al.  Online video synthesis for removing occluding objects using multiple uncalibrated cameras via plane sweep algorithm , 2009, 2009 Third ACM/IEEE International Conference on Distributed Smart Cameras (ICDSC).

[4]  Takeo Kanade,et al.  Shape reconstruction in projective grid space from large number of images , 1999, Proceedings. 1999 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (Cat. No PR00149).

[5]  Steve Mann,et al.  Mediated Reality , 1999 .

[6]  Hideo Saito,et al.  Real-time free viewpoint video from uncalibrated cameras using plane-sweep algorithm , 2009, 2009 IEEE 12th International Conference on Computer Vision Workshops, ICCV Workshops.

[7]  Steven M. Seitz,et al.  Physically-valid view synthesis by image interpolation , 1995, Proceedings IEEE Workshop on Representation of Visual Scenes (In Conjunction with ICCV'95).

[8]  Ruigang Yang,et al.  Real‐Time Consensus‐Based Scene Reconstruction Using Commodity Graphics Hardware † , 2003, Comput. Graph. Forum.

[9]  Robert T. Collins,et al.  A space-sweep approach to true multi-image matching , 1996, Proceedings CVPR IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[10]  Steven M. Seitz,et al.  View morphing , 1996, SIGGRAPH.

[11]  Ronald Azuma,et al.  A Survey of Augmented Reality , 1997, Presence: Teleoperators & Virtual Environments.

[12]  Lance Williams,et al.  View Interpolation for Image Synthesis , 1993, SIGGRAPH.

[13]  Ronald Azuma,et al.  Recent Advances in Augmented Reality , 2001, IEEE Computer Graphics and Applications.

[14]  Nassir Navab,et al.  Multiview paraperspective projection model for diminished reality , 2003, The Second IEEE and ACM International Symposium on Mixed and Augmented Reality, 2003. Proceedings..

[15]  Bernhard P. Wrobel,et al.  Multiple View Geometry in Computer Vision , 2001 .

[16]  Hideo Saito,et al.  Diminished Reality Via Multiple Hand-Held CAMERAS , 2007, 2007 First ACM/IEEE International Conference on Distributed Smart Cameras.

[17]  Didier Arquès,et al.  Real-time Plane-sweep with Local Strategy , 2006, J. WSCG.

[18]  Steve Mann,et al.  A parallel mediated reality platform , 2004, 2004 International Conference on Image Processing, 2004. ICIP '04..