Kinect depth map inpainting using spline approximation

Image inpainting consists in reconstructing missing parts of a given image. In this work, we propose to use an approximation method based on splines and finite elements approximation to recover missing depth values in images acquired with a Kinect 3D sensor. Neighboring pixels in the depth map may contain very different distance values. The considered surface approximation problem therefore involves rapidly varying data which can lead to oscillations (Gibbs phenomenon). To address this issue, we propose to apply two scale transformations to dampen these oscillations near steep gradients implied by the data. The algorithm is presented with some numerical examples of inpainting. Our approach also permits to get a finer resolution of the 3D depth map.

[1]  Tony F. Chan,et al.  Euler's Elastica and Curvature-Based Inpainting , 2003, SIAM J. Appl. Math..

[2]  Daniel Cremers,et al.  A Linear Framework for Region-Based Image Segmentation and Inpainting Involving Curvature Penalization , 2011, International Journal of Computer Vision.

[3]  Christian Gout,et al.  A result about scale transformation families in approximation: application to surface fitting from rapidly varying data , 2004, Numerical Algorithms.

[4]  Luis Salgado,et al.  Efficient spatio-temporal hole filling strategy for Kinect depth maps , 2012, Electronic Imaging.

[5]  Saïd Ladjal,et al.  Exemplar-Based Inpainting from a Variational Point of View , 2010, SIAM J. Math. Anal..

[6]  Juho Kannala,et al.  Depth Map Inpainting under a Second-Order Smoothness Prior , 2013, SCIA.

[7]  Andrea L. Bertozzi,et al.  A Wavelet-Laplace Variational Technique for Image Deconvolution and Inpainting , 2008, IEEE Transactions on Image Processing.

[8]  María Cruz López de Silanes,et al.  Multidimensional Minimizing Splines: Theory and Applications , 2004 .

[9]  Andrew S. Glassner,et al.  Proceedings of the 27th annual conference on Computer graphics and interactive techniques , 1994, SIGGRAPH 1994.

[10]  Guangming Shi,et al.  Structure guided fusion for depth map inpainting , 2013, Pattern Recognit. Lett..

[11]  Patrick Pérez,et al.  Geometrically Guided Exemplar-Based Inpainting , 2011, SIAM J. Imaging Sci..

[12]  Aberrahman Bouhamidi.  Interpolation et approximation par des fonctions splines radiales a plusieurs variables , 1992 .

[13]  C. R. Deboor,et al.  A practical guide to splines , 1978 .

[14]  Richard Franke,et al.  Thin plate splines with tension , 1985, Comput. Aided Geom. Des..

[15]  Andrea L. Bertozzi,et al.  Inpainting of Binary Images Using the Cahn–Hilliard Equation , 2007, IEEE Transactions on Image Processing.

[16]  Marcelo Bertalmío,et al.  FLUID DYNAMICS, AND IMAGE AND VIDEO INPAINTING , 2001 .

[17]  Bruno Portier,et al.  A New Spherical Mixture Model for Head Detection in Depth Images , 2014, SIAM J. Imaging Sci..

[18]  Jitendra Malik,et al.  Scale-Space and Edge Detection Using Anisotropic Diffusion , 1990, IEEE Trans. Pattern Anal. Mach. Intell..

[19]  Carl de Boor,et al.  A Practical Guide to Splines , 1978, Applied Mathematical Sciences.

[20]  Jean-Michel Morel,et al.  Level lines based disocclusion , 1998, Proceedings 1998 International Conference on Image Processing. ICIP98 (Cat. No.98CB36269).

[21]  N. Aronszajn.  Theory of Reproducing Kernels. , 1950 .

[22]  Philippe G. Ciarlet,et al.  The finite element method for elliptic problems , 2002, Classics in applied mathematics.

[23]  L. Rudin,et al.  Nonlinear total variation based noise removal algorithms , 1992 .

[24]  Haibo Wang,et al.  Kinect depth restoration via energy minimization with TV21 regularization , 2013, 2013 IEEE International Conference on Image Processing.

[25]  Christian Gout,et al.  Ck surface approximation from surface patches , 2002 .

[26]  Christian Gout,et al.  An algorithm for automatic people detection from depth map sequences , 2014, 2014 5th European Workshop on Visual Information Processing (EUVIP).

[27]  Guillermo Sapiro,et al.  Simultaneous structure and texture image inpainting , 2003, IEEE Trans. Image Process..

[28]  Simon Masnou,et al.  Disocclusion: a variational approach using level lines , 2002, IEEE Trans. Image Process..

[29]  Guillermo Sapiro,et al.  Navier-stokes, fluid dynamics, and image and video inpainting , 2001, Proceedings of the 2001 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. CVPR 2001.

[30]  I. J. Schoenberg.  Contributions to the problem of approximation of equidistant data by analytic functions. Part B. On the problem of osculatory interpolation. A second class of analytic approximation formulae , 1946 .