The ORIGAMI Project: Advanced tools for creating and mixing real and virtual content in film and TV production

The goal of the EC-funded IST project ORIGAMI was the development of advanced tools and new production techniques for high-quality and seamless integration of real and virtual content in film and TV productions. In particular, the project focused on pre-production tools for automatic virtual set creation through image-based 3-D modelling of environments and objects. One key goal of the project was to achieve real-time in-studio pre-visualisation of the virtual elements (objects and actors) of the extended set. In this contribution the studio pre-visualisation system that has been developed for the project is illustrated, and its usage within a high-end film and TV production environment is described. Furthermore an overview of the developed solutions for automatic generation of 3-D models of environments, static objects and (moving) actors is given.

[1]  Benjamin B. Kimia,et al.  A formal classification of 3D medial axis points and their local geometry , 2000, Proceedings IEEE Conference on Computer Vision and Pattern Recognition. CVPR 2000 (Cat. No.PR00662).

[2]  Alex Pentland,et al.  Recursive Estimation of Motion, Structure, and Focal Length , 1995, IEEE Trans. Pattern Anal. Mach. Intell..

[3]  Reinhard Koch,et al.  Image Based Interactive Rendering with View Dependent Geometry , 2003, Comput. Graph. Forum.

[4]  Reinhard Koch,et al.  Self-Calibration and Metric Reconstruction Inspite of Varying and Unknown Intrinsic Camera Parameters , 1999, International Journal of Computer Vision.

[5]  A. Sarti,et al.  A fast level-set approach to 2D and 3D reconstruction from unorganized sample points , 2003, 3rd International Symposium on Image and Signal Processing and Analysis, 2003. ISPA 2003. Proceedings of the.

[6]  Reinhard Koch,et al.  Multi Viewpoint Stereo from Uncalibrated Video Sequences , 1998, ECCV.

[7]  Reinhard Koch,et al.  Multi-camera real-time depth estimation with discontinuity handling on PC graphics hardware , 2004, Proceedings of the 17th International Conference on Pattern Recognition, 2004. ICPR 2004..

[8]  Reinhard Koch,et al.  Plenoptic Modeling of 3D Scenes with a Sensor-augmented Multi-Camera Rig , 2002 .

[9]  Stefano Soatto,et al.  3-D Motion and Structure from 2-D Motion Causally Integrated over Time: Implementation , 2000, ECCV.

[10]  Paul A. Beardsley,et al.  3D Model Acquisition from Extended Image Sequences , 1996, ECCV.

[11]  Richard I. Hartley,et al.  Estimation of Relative Camera Positions for Uncalibrated Cameras , 1992, ECCV.

[12]  Bernhard P. Wrobel,et al.  Multiple View Geometry in Computer Vision , 2001 .

[13]  Stefano Soatto,et al.  Real-time 3D motion and structure of point features: a front-end system for vision-based control and interaction , 2000, Proceedings IEEE Conference on Computer Vision and Pattern Recognition. CVPR 2000 (Cat. No.PR00662).

[14]  Oliver Grau,et al.  A combined studio production system for 3-D capturing of live action and immersive actor feedback , 2004, IEEE Transactions on Circuits and Systems for Video Technology.

[15]  Vladimir Kolmogorov,et al.  Multi-camera Scene Reconstruction via Graph Cuts , 2002, ECCV.

[16]  Oliver Grau,et al.  3D image sequence acquisition for TV & film production , 2002, Proceedings. First International Symposium on 3D Data Processing Visualization and Transmission.

[17]  Ingemar J. Cox,et al.  A Maximum Likelihood Stereo Algorithm , 1996, Comput. Vis. Image Underst..

[18]  Alex Pentland,et al.  3D structure from 2D motion , 1999, IEEE Signal Process. Mag..

[19]  Francoise Preteux,et al.  MPEG-4 Human Virtual Body Animation , 2002 .

[20]  Reinhard Koch,et al.  Real-time multi-stereo depth estimation on GPU with approximative discontinuity handling , 2004 .

[21]  Reinhard Koch,et al.  Self-calibration and metric reconstruction in spite of varying and unknown internal camera parameters , 1998, Sixth International Conference on Computer Vision (IEEE Cat. No.98CH36271).

[22]  James A. Sethian,et al.  Level Set Methods and Fast Marching Methods , 1999 .

[23]  Augusto Sarti,et al.  Estimation of radiometric parameters for a realistic rendering of 3D models , 1999, Proceedings 1999 International Conference on Image Processing (Cat. 99CH36348).

[24]  Lutz Falkenhagen.  Hierarchical Block-Based Disparity Estimation Considering Neighbourhood Constraints , 1997 .

[25]  Olivier D. Faugeras,et al.  Variational principles, surface evolution, PDEs, level set methods, and the stereo problem , 1998, IEEE Trans. Image Process..

[26]  Fabio Lavagetto,et al.  The facial animation engine: toward a high-level interface for the design of MPEG-4 compliant animated faces , 1999, IEEE Trans. Circuits Syst. Video Technol..

[27]  Alex Pentland,et al.  3 D Structure from 2 D Motion , 2001 .

[28]  B. Barsky,et al.  An Introduction to Splines for Use in Computer Graphics and Geometric Modeling , 1987 .

[29]  John C. Hart,et al.  The CAVE: audio visual experience automatic virtual environment , 1992, CACM.

[30]  Oliver Grau,et al.  Use of 3D techniques for virtual production , 2000, IS&T/SPIE Electronic Imaging.

[31]  P. Perona,et al.  Motion estimation via dynamic vision , 1994, Proceedings of 1994 33rd IEEE Conference on Decision and Control.

[32]  P PentlandAlex,et al.  Recursive Estimation of Motion, Structure, and Focal Length , 1995 .

[33]  Augusto Sarti,et al.  Analysis of Image Sequences for 3 D Camera Motion Estimation , 2001 .

[34]  Oliver Grau,et al.  A FAST AND ACCURATE METHOD FOR 3D SURFACE RECONSTRUCTION FROM IMAGE SILHOUETTES , 2003 .

[35]  Reinhard Koch,et al.  Calibration of hand-held camera sequences for plenoptic modeling , 1999, Proceedings of the Seventh IEEE International Conference on Computer Vision.

[36]  Fabio Lavagetto,et al.  The Facial Animation Engine , 2003 .

[37]  Richard Szeliski,et al.  A Taxonomy and Evaluation of Dense Two-Frame Stereo Correspondence Algorithms , 2001, International Journal of Computer Vision.

[38]  Ruigang Yang,et al.  Multi-resolution real-time stereo on commodity graphics hardware , 2003, 2003 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2003. Proceedings..

[39]  G. A. Thomas,et al.  A versatile camera position measurement system for virtual reality TV production , 1997 .

[40]  Augusto Sarti,et al.  Image-Based Multiresolution Implicit Object Modeling , 2002, EURASIP J. Adv. Signal Process..

[41]  James A. Sethian,et al.  Level Set Methods and Fast Marching Methods: Evolving Interfaces in Computational Geometry, Fluid , 2012 .

[42]  Oliver Grau.  3D sequence generation from multiple cameras , 2004, IEEE 6th Workshop on Multimedia Signal Processing, 2004..