Objective Quality Assessment in Free-Viewpoint Video Production

This paper addresses the problem of objectively measuring quality in free-viewpoint video production. The accuracy of scene reconstruction is typically limited and an evaluation of free-viewpoint video should explicitly consider the quality of image production. A simple objective measure of accuracy is presented in terms of structural registration error in view synthesis. This technique can be applied as a full-reference metric to measure the fidelity of view synthesis to a ground truth image or as a no-reference metric to measure the error in registering scene appearance in image-based rendering. The metric is applied to a data-set with known geometric accuracy and a comparison is also demonstrated between two free-viewpoint video techniques across two prototype production studios.

[1]  Alan C. Bovik,et al.  A Structural Similarity Metric for Video Based on Motion Models , 2007, 2007 IEEE International Conference on Acoustics, Speech and Signal Processing - ICASSP '07.

[2]  Andrew P. Bradley,et al.  Perceptual quality metrics applied to still image compression , 1998, Signal Process..

[3]  Richard Szeliski,et al.  Rapid octree construction from image sequences , 1993 .

[4]  William E. Lorensen,et al.  Marching cubes: a high resolution 3D surface construction algorithm , 1996 .

[5]  Pascual Capilla,et al.  Image quality metric based on multidimensional contrast perception models , 1999 .

[6]  Jean Ponce,et al.  Carved Visual Hulls for Image-Based Modeling , 2006, International Journal of Computer Vision.

[7]  Tiago Rosa Maria Paula Queluz,et al.  Towards Objective Metrics for Blind Assessment of Images Quality , 2006, 2006 International Conference on Image Processing.

[8]  Francis Schmitt,et al.  Silhouette and stereo fusion for 3D object modeling , 2003, Fourth International Conference on 3-D Digital Imaging and Modeling, 2003. 3DIM 2003. Proceedings..

[9]  A. Laurentini,et al.  The Visual Hull Concept for Silhouette-Based Image Understanding , 1994, IEEE Trans. Pattern Anal. Mach. Intell..

[10]  Kiriakos N. Kutulakos,et al.  A Theory of Shape by Space Carving , 2000, International Journal of Computer Vision.

[11]  Jérémie Allard,et al.  The GrImage Platform: A Mixed Reality Environment for Interactions , 2006, Fourth IEEE International Conference on Computer Vision Systems (ICVS'06).

[12]  Adrian Hilton,et al.  Surface Capture for Performance-Based Animation , 2007, IEEE Computer Graphics and Applications.

[13]  Takeo Kanade,et al.  Virtualized Reality: Constructing Virtual Worlds from Real Scenes , 1997, IEEE Multim..

[14]  Bülent Sankur,et al.  Statistical evaluation of image quality measures , 2002, J. Electronic Imaging.

[15]  Kiriakos N. Kutulakos.  Approximate N-View Stereo , 2000, ECCV.

[16]  Xiaojun Wu,et al.  Real-time 3D shape reconstruction, dynamic 3D mesh deformation, and high fidelity visualization for 3D video , 2004, Comput. Vis. Image Underst..

[17]  Andrew W. Fitzgibbon,et al.  Efficient new-view synthesis using pairwise dictionary priors , 2007, 2007 IEEE Conference on Computer Vision and Pattern Recognition.

[18]  Ventseslav Sainov,et al.  3-D Time-Varying Scene Capture Technologies—A Survey , 2007, IEEE Transactions on Circuits and Systems for Video Technology.

[19]  Oliver Grau,et al.  Dual-Mode Deformable Models for Free-Viewpoint Video of Sports Events , 2007, Sixth International Conference on 3-D Digital Imaging and Modeling (3DIM 2007).

[20]  William J. Christmas,et al.  Filtering requirements for gradient-based optical flow measurement , 2000, IEEE Trans. Image Process..

[21]  Adrian Hilton,et al.  A Comparative Study of Free-Viewpoint Video Techniques For sports events , 2006 .

[22]  David Salesin,et al.  A Bayesian approach to digital matting , 2001, Proceedings of the 2001 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. CVPR 2001.

[23]  Katsushi Ikeuchi,et al.  Microfacet Billboarding , 2002, Rendering Techniques.

[24]  Takeo Kanade,et al.  Image-based spatio-temporal modeling and view interpolation of dynamic events , 2005, TOGS.

[25]  Marc Levoy,et al.  Light field rendering , 1996, SIGGRAPH.

[26]  Tomás Svoboda,et al.  A Convenient Multicamera Self-Calibration for Virtual Environments , 2005, Presence: Teleoperators & Virtual Environments.

[27]  I. Kitahara,et al.  Live mixed-reality 3D video in soccer stadium , 2003, The Second IEEE and ACM International Symposium on Mixed and Augmented Reality, 2003. Proceedings..

[28]  Roberto Cipolla,et al.  Multiview Stereo via Volumetric Graph-Cuts and Occlusion Robust Photo-Consistency , 2007, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[29]  Ian D. Reid,et al.  A Multiple View Layered Representation for Dynamic Novel View Synthesis , 2003, BMVC.

[30]  Adrian Hilton,et al.  Exact View-Dependent Visual Hulls , 2006, 18th International Conference on Pattern Recognition (ICPR'06).

[31]  Adrian Hilton,et al.  A Free-Viewpoint Video System for Visualization of Sport Scenes , 2007 .

[32]  Alan C. Bovik,et al.  Image information and visual quality , 2006, IEEE Trans. Image Process..

[33]  Yizhou Yu,et al.  Efficient View-Dependent Image-Based Rendering with Projective Texture-Mapping , 1998, Rendering Techniques.

[34]  Adrian Hilton,et al.  Wand-based Multiple Camera Studio Calibration , 2007 .

[35]  Takeo Kanade,et al.  A real time system for robust 3D voxel reconstruction of human motions , 2000, Proceedings IEEE Conference on Computer Vision and Pattern Recognition. CVPR 2000 (Cat. No.PR00662).

[36]  Eero P. Simoncelli,et al.  Image quality assessment: from error visibility to structural similarity , 2004, IEEE Transactions on Image Processing.

[37]  Andreas Kunz,et al.  blue-c: a spatially immersive display and 3D video portal for telepresence , 2003, ACM Trans. Graph..

[38]  Thrasyvoulos N. Pappas,et al.  Perceptual criteria for image quality evaluation , 2005 .

[39]  Hideo Saito,et al.  Arbitrary viewpoint observation for soccer match video , 2004 .

[40]  Hans-Peter Seidel,et al.  Free-viewpoint video of human actors , 2003, ACM Trans. Graph..

[41]  M. Magnor,et al.  Space-time isosurface evolution for temporally coherent 3D reconstruction , 2004, CVPR 2004.

[42]  Michael Bosse,et al.  Unstructured lumigraph rendering , 2001, SIGGRAPH.

[43]  Richard Szeliski,et al.  A Comparison and Evaluation of Multi-View Stereo Reconstruction Algorithms , 2006, 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'06).

[44]  Markus H. Gross,et al.  3D Video Billboard Clouds , 2007, Comput. Graph. Forum.

[45]  Oliver Grau,et al.  A Bayesian Framework for Simultaneous Matting and 3D Reconstruction , 2007, Sixth International Conference on 3-D Digital Imaging and Modeling (3DIM 2007).

[46]  Wojciech Matusik,et al.  Polyhedral Visual Hulls for Real-Time Rendering , 2001, Rendering Techniques.

[47]  Stefan Winkler.  Video quality and beyond , 2007, 2007 15th European Signal Processing Conference.

[48]  Hans-Peter Seidel,et al.  Performance capture from sparse multi-view video , 2008, SIGGRAPH 2008.

[49]  Wojciech Matusik,et al.  3D TV: a scalable system for real-time acquisition, transmission, and autostereoscopic display of dynamic scenes , 2004, ACM Trans. Graph..

[50]  Paul S. Fisher,et al.  Image quality measures and their performance , 1995, IEEE Trans. Commun..

[51]  Wojciech Matusik,et al.  Articulated mesh animation from multi-view silhouettes , 2008, ACM Trans. Graph..

[52]  Stefan Winkler,et al.  Perceptual distortion metric for digital color video , 1999, Electronic Imaging.

[53]  Oliver Grau,et al.  A combined studio production system for 3-D capturing of live action and immersive actor feedback , 2004, IEEE Transactions on Circuits and Systems for Video Technology.

[54]  Alessandro Neri,et al.  A comparison between an objective quality measure and the mean annoyance values of watermarked videos , 2002, Proceedings. International Conference on Image Processing.

[55]  Richard Szeliski,et al.  High-quality video view interpolation using a layered representation , 2004, SIGGRAPH 2004.

[56]  Adrian Hilton,et al.  Model-based multiple view reconstruction of people , 2003, Proceedings Ninth IEEE International Conference on Computer Vision.