Real-Time View Correction for Mobile Devices

We present a real-time method for rendering novel virtual camera views from given RGB-D (color and depth) data of a different viewpoint. Missing color and depth information due to incomplete input or disocclusions is efficiently inpainted in a temporally consistent way. The inpainting takes the location of strong image gradients into account as likely depth discontinuities. We present our method in the context of a view correction system for mobile devices, and discuss how to obtain a screen-camera calibration and options for acquiring depth input. Our method has use cases in both augmented and virtual reality applications. We demonstrate the speed of our system and the visual quality of its results in multiple experiments in the paper as well as in the supplementary video.

[1]  Gerhard Rigoll,et al.  Depth Inpainting with Tensor Voting using Local Geometry , 2012, VISAPP.

[2]  Stergios I. Roumeliotis,et al.  A Square Root Inverse Filter for Efficient Vision-aided Inertial Navigation on Mobile Devices , 2015, Robotics: Science and Systems.

[3]  David Tschumperlé,et al.  Superpixel-based depth map inpainting for RGB-D view synthesis , 2015, 2015 IEEE International Conference on Image Processing (ICIP).

[4]  Xavier Bresson,et al.  Fast Global Minimization of the Active Contour/Snake Model , 2007, Journal of Mathematical Imaging and Vision.

[5]  Antonin Chambolle,et al.  Diagonal preconditioning for first order primal-dual algorithms in convex optimization , 2011, 2011 International Conference on Computer Vision.

[6]  Yong Man Ro,et al.  Inter-view consistent hole filling in view extrapolation for multi-view image generation , 2014, 2014 IEEE International Conference on Image Processing (ICIP).

[7]  Naokazu Yokoya,et al.  Diminished Reality Based on Image Inpainting Considering Background Geometry , 2016, IEEE Transactions on Visualization and Computer Graphics.

[8]  David Tschumperlé,et al.  Depth-Guided Disocclusion Inpainting of Synthesized RGB-D Images , 2017, IEEE Transactions on Image Processing.

[9]  Marc Pollefeys,et al.  Two Cameras and a Screen: How to Calibrate Mobile Devices? , 2014, 2014 2nd International Conference on 3D Vision.

[10]  Antonin Chambolle,et al.  A First-Order Primal-Dual Algorithm for Convex Problems with Applications to Imaging , 2011, Journal of Mathematical Imaging and Vision.

[11]  Manuel Menezes de Oliveira Neto,et al.  Fast Digital Image Inpainting , 2001, VIIP.

[12]  Luiz Fernando Gomes Soares,et al.  Real-Time Depth-Image-Based Rendering for 3DTV Using OpenCL , 2014, ISVC.

[13]  Steven M. Seitz,et al.  View morphing , 1996, SIGGRAPH.

[14]  Anita Sellent,et al.  Virtual Video Camera: Image‐Based Viewpoint Navigation Through Space and Time , 2010, Comput. Graph. Forum.

[15]  Tony F. Chan,et al.  Mathematical Models for Local Nontexture Inpaintings , 2002, SIAM J. Appl. Math..

[16]  Siddhartha S. Srinivasa,et al.  Chisel: Real Time Large Scale 3D Reconstruction Onboard a Mobile Device using Spatially Hashed Signed Distance Fields , 2015, Robotics: Science and Systems.

[17]  Zhengyou Zhang,et al.  A Flexible New Technique for Camera Calibration , 2000, IEEE Trans. Pattern Anal. Mach. Intell..

[18]  Marc Levoy,et al.  A volumetric method for building complex models from range images , 1996, SIGGRAPH.

[19]  Dimitrios G. Kottas,et al.  Camera-IMU-based localization: Observability analysis and consistency improvement , 2014, Int. J. Robotics Res..

[20]  G. M.,et al.  Partial Differential Equations I , 2023, Applied Mathematical Sciences.

[21]  Kosuke Sato,et al.  Approximated user-perspective rendering in tablet-based augmented reality , 2013, 2013 IEEE International Symposium on Mixed and Augmented Reality (ISMAR).

[22]  Marcus A. Magnor,et al.  Multi-image interpolation based on graph-cuts and symmetric optical flow , 2010, SIGGRAPH '10.

[23]  William E. Lorensen,et al.  Marching cubes: A high resolution 3D surface construction algorithm , 1987, SIGGRAPH.

[24]  Tobias Höllerer,et al.  User-perspective augmented reality magic lens from gradients , 2014, VRST '14.

[25]  Michael Bosse,et al.  Get Out of My Lab: Large-scale, Real-Time Visual-Inertial Localization , 2015, Robotics: Science and Systems.

[26]  Matthias Nießner,et al.  Real-time 3D reconstruction at scale using voxel hashing , 2013, ACM Trans. Graph..

[27]  Matthew Turk,et al.  User-Perspective AR Magic Lens from Gradient-Based IBR and Semi-Dense Stereo , 2017, IEEE Transactions on Visualization and Computer Graphics.

[28]  Janne Heikkilä,et al.  A four-step camera calibration procedure with implicit image correction , 1997, Proceedings of IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[29]  Patrick Pérez,et al.  Region filling and object removal by exemplar-based image inpainting , 2004, IEEE Transactions on Image Processing.

[30]  Yuesheng Zhu,et al.  A Hole Filling Approach Based on Background Reconstruction for View Synthesis in 3D Video , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[31]  King Ngi Ngan,et al.  Screen-camera calibration using a thread , 2014, 2014 IEEE International Conference on Image Processing (ICIP).

[32]  Lance Williams,et al.  View Interpolation for Image Synthesis , 1993, SIGGRAPH.

[33]  Olaf Kähler,et al.  Very High Frame Rate Volumetric Integration of Depth Images on Mobile Devices , 2015, IEEE Transactions on Visualization and Computer Graphics.

[34]  Richard Szeliski,et al.  High-quality video view interpolation using a layered representation , 2004, SIGGRAPH 2004.