Objective Quality Assessment in Free-Viewpoint Video Production

This paper addresses the problem of objectively measuring quality in free-viewpoint video production. The accuracy of scene reconstruction is typically limited and an evaluation of free-viewpoint video should explicitly consider the quality of image production. A simple objective measure of accuracy is presented in terms of structural registration error in view synthesis. This technique can be applied as a full-reference metric to measure the fidelity of view synthesis to a ground truth image or as a no-reference metric to measure the error in registering scene appearance in image-based rendering. The metric is applied to a data-set with known geometric accuracy and a comparison is also demonstrated between two free-viewpoint video techniques across two prototype production studios.

[1]  Michael Bosse,et al.  Unstructured lumigraph rendering , 2001, SIGGRAPH.

[2]  Richard Szeliski,et al.  High-quality video view interpolation using a layered representation , 2004, SIGGRAPH 2004.

[3]  Luc Van Gool,et al.  Blue-c: a spatially immersive display and 3D video portal for telepresence , 2003, IPT/EGVE.

[4]  Stefan Bilbao,et al.  Proceedings of the European Signal Processing Conference , 2005 .

[5]  Richard Szeliski,et al.  A Comparison and Evaluation of Multi-View Stereo Reconstruction Algorithms , 2006, 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'06).

[6]  William E. Lorensen,et al.  Marching cubes: A high resolution 3D surface construction algorithm , 1987, SIGGRAPH.

[7]  Markus H. Gross,et al.  3D Video Billboard Clouds , 2007, Comput. Graph. Forum.

[8]  Oliver Grau,et al.  A Bayesian Framework for Simultaneous Matting and 3D Reconstruction , 2007, Sixth International Conference on 3-D Digital Imaging and Modeling (3DIM 2007).

[9]  Marc Levoy,et al.  Light field rendering , 1996, SIGGRAPH.

[10]  Wojciech Matusik,et al.  3D TV: a scalable system for real-time acquisition, transmission, and autostereoscopic display of dynamic scenes , 2004, ACM Trans. Graph..

[11]  Xiaojun Wu,et al.  Real-time 3D shape reconstruction, dynamic 3D mesh deformation, and high fidelity visualization for 3D video , 2004, Comput. Vis. Image Underst..

[12]  Hans-Peter Seidel,et al.  Free-viewpoint video of human actors , 2003, ACM Trans. Graph..

[13]  Andreas Kunz,et al.  blue-c: a spatially immersive display and 3D video portal for telepresence , 2003, ACM Trans. Graph..

[14]  Oliver Grau,et al.  Dual-Mode Deformable Models for Free-Viewpoint Video of Sports Events , 2007, Sixth International Conference on 3-D Digital Imaging and Modeling (3DIM 2007).

[15]  Stefan Winkler.  Video quality and beyond , 2007, 2007 15th European Signal Processing Conference.

[16]  Adrian Hilton,et al.  Model-based multiple view reconstruction of people , 2003, Proceedings Ninth IEEE International Conference on Computer Vision.

[17]  William J. Christmas,et al.  Filtering requirements for gradient-based optical flow measurement , 2000, IEEE Trans. Image Process..

[18]  Eero P. Simoncelli,et al.  Image quality assessment: from error visibility to structural similarity , 2004, IEEE Transactions on Image Processing.

[19]  Jean Ponce,et al.  Carved Visual Hulls for Image-Based Modeling , 2006, International Journal of Computer Vision.

[20]  Francis Schmitt,et al.  Silhouette and stereo fusion for 3D object modeling , 2003, Fourth International Conference on 3-D Digital Imaging and Modeling, 2003. 3DIM 2003. Proceedings..

[21]  Andrew W. Fitzgibbon,et al.  Efficient new-view synthesis using pairwise dictionary priors , 2007, 2007 IEEE Conference on Computer Vision and Pattern Recognition.

[22]  David Salesin,et al.  A Bayesian approach to digital matting , 2001, Proceedings of the 2001 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. CVPR 2001.

[23]  Katsushi Ikeuchi,et al.  Microfacet Billboarding , 2002, Rendering Techniques.

[24]  Takeo Kanade,et al.  Image-based spatio-temporal modeling and view interpolation of dynamic events , 2005, TOGS.

[25]  Jérémie Allard,et al.  The GrImage Platform: A Mixed Reality Environment for Interactions , 2006, Fourth IEEE International Conference on Computer Vision Systems (ICVS'06).

[26]  I. Kitahara,et al.  Live mixed-reality 3D video in soccer stadium , 2003, The Second IEEE and ACM International Symposium on Mixed and Augmented Reality, 2003. Proceedings..

[27]  Francis Schmitt,et al.  Silhouette and stereo fusion for 3D object modeling , 2003, Fourth International Conference on 3-D Digital Imaging and Modeling, 2003. 3DIM 2003. Proceedings..

[28]  Paul S. Fisher,et al.  Image quality measures and their performance , 1995, IEEE Trans. Commun..

[29]  Adrian Hilton,et al.  A Free-Viewpoint Video System for Visualization of Sport Scenes , 2007 .

[30]  Tomás Svoboda,et al.  A Convenient Multicamera Self-Calibration for Virtual Environments , 2005, Presence: Teleoperators & Virtual Environments.

[31]  Hans-Peter Seidel,et al.  Performance capture from sparse multi-view video , 2008, ACM Trans. Graph..

[32]  Takeo Kanade,et al.  Virtualized Reality: Constructing Virtual Worlds from Real Scenes , 1997, IEEE Multim..

[33]  Anita Sellent,et al.  Floating Textures , 2008, Comput. Graph. Forum.

[34]  Adrian Hilton,et al.  Exact View-Dependent Visual Hulls , 2006, 18th International Conference on Pattern Recognition (ICPR'06).

[35]  Richard Szeliski,et al.  Rapid octree construction from image sequences , 1993 .

[36]  Roberto Cipolla,et al.  Multiview Stereo via Volumetric Graph-Cuts and Occlusion Robust Photo-Consistency , 2007, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[37]  Ian D. Reid,et al.  A Multiple View Layered Representation for Dynamic Novel View Synthesis , 2003, BMVC.

[38]  Wojciech Matusik,et al.  Polyhedral Visual Hulls for Real-Time Rendering , 2001, Rendering Techniques.

[39]  Adrian Hilton,et al.  Surface Capture for Performance-Based Animation , 2007, IEEE Computer Graphics and Applications.

[40]  Alan C. Bovik,et al.  Image information and visual quality , 2004, 2004 IEEE International Conference on Acoustics, Speech, and Signal Processing.

[41]  Yizhou Yu,et al.  Efficient View-Dependent Image-Based Rendering with Projective Texture-Mapping , 1998, Rendering Techniques.

[42]  Adrian Hilton,et al.  Wand-based Multiple Camera Studio Calibration , 2007 .

[43]  Roberto Scopigno,et al.  Computer Graphics forum , 2003, Computer Graphics Forum.

[44]  Alan C. Bovik,et al.  A Structural Similarity Metric for Video Based on Motion Models , 2007, 2007 IEEE International Conference on Acoustics, Speech and Signal Processing - ICASSP '07.

[45]  Andrew P. Bradley,et al.  Perceptual quality metrics applied to still image compression , 1998, Signal Process..

[46]  Pascual Capilla,et al.  Image quality metric based on multidimensional contrast perception models , 1999 .

[47]  Tiago Rosa Maria Paula Queluz,et al.  Towards Objective Metrics for Blind Assessment of Images Quality , 2006, 2006 International Conference on Image Processing.

[48]  Bülent Sankur,et al.  Statistical evaluation of image quality measures , 2002, J. Electronic Imaging.

[49]  Adrian Hilton,et al.  Objective Quality Assessment in Free-Viewpoint Video Production , 2008, 3DTV-CON 2008.

[50]  Adrian Hilton,et al.  A Comparative Study of Free-Viewpoint Video Techniques For sports events , 2006 .

[51]  A. Laurentini,et al.  The Visual Hull Concept for Silhouette-Based Image Understanding , 1994, IEEE Trans. Pattern Anal. Mach. Intell..

[52]  Kiriakos N. Kutulakos,et al.  A Theory of Shape by Space Carving , 2000, International Journal of Computer Vision.

[53]  Alessandro Neri,et al.  A comparison between an objective quality measure and the mean annoyance values of watermarked videos , 2002, Proceedings. International Conference on Image Processing.

[54]  Ventseslav Sainov,et al.  3-D Time-Varying Scene Capture Technologies—A Survey , 2007, IEEE Transactions on Circuits and Systems for Video Technology.

[56]  Kiriakos N. Kutulakos.  Approximate N-View Stereo , 2000, ECCV.

[57]  Takeo Kanade,et al.  A real time system for robust 3D voxel reconstruction of human motions , 2000, Proceedings IEEE Conference on Computer Vision and Pattern Recognition. CVPR 2000 (Cat. No.PR00662).

[58]  Thrasyvoulos N. Pappas,et al.  Perceptual criteria for image quality evaluation , 2005 .

[59]  Hideo Saito,et al.  Arbitrary viewpoint observation for soccer match video , 2004 .

[60]  Marcus A. Magnor,et al.  Space-time isosurface evolution for temporally coherent 3D reconstruction , 2004, Proceedings of the 2004 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2004. CVPR 2004..

[61]  Wojciech Matusik,et al.  Articulated mesh animation from multi-view silhouettes , 2008, ACM Trans. Graph..

[62]  Stefan Winkler,et al.  Perceptual distortion metric for digital color video , 1999, Electronic Imaging.

[63]  Oliver Grau,et al.  A combined studio production system for 3-D capturing of live action and immersive actor feedback , 2004, IEEE Transactions on Circuits and Systems for Video Technology.