A sparsity-promoting image decomposition model for depth recovery

Abstract This paper proposes a novel image decomposition model for scene depth recovery from low-quality depth measurements and its corresponding high resolution color image. Through our observation, the depth map mainly contains smooth regions separated by additive step discontinuities, and can be simultaneously decomposed into a local smooth surface and an approximately piecewise constant component. Therefore, the proposed unified model combines the least square polynomial approximation (for smooth surface) and a sparsity-promoting prior (for piecewise constant) to better portray the 2D depth signal intrinsically. As we know, the representation of the piecewise constant signal in gradient domain is extremely sparse. Previous researches using total variation filter based on L1-norm or Lp-norm (0

[1]  Kwanghoon Sohn,et al.  Structure Selective Depth Superresolution for RGB-D Cameras , 2016, IEEE Transactions on Image Processing.

[2]  Pascal Frossard,et al.  Global Auto-Regressive Depth Recovery via Iterative Non-Local Filtering , 2019, IEEE Transactions on Broadcasting.

[3]  Michael S. Brown,et al.  High quality depth map upsampling for 3D-TOF cameras , 2011, 2011 International Conference on Computer Vision.

[4]  Qiang Wu,et al.  Explicit Edge Inconsistency Evaluation Model for Color-Guided Depth Map Enhancement , 2018, IEEE Transactions on Circuits and Systems for Video Technology.

[5]  Sebastian Thrun,et al.  An Application of Markov Random Fields to Range Sensing , 2005, NIPS.

[6]  Björn E. Ottersten,et al.  Real-time depth enhancement by fusion for RGB-D cameras , 2013, IET Comput. Vis..

[7]  Shiu Yin Yuen,et al.  Illumination direction estimation for augmented reality using a surface input real valued output regression network , 2010, Pattern Recognit..

[8]  Robert D. Nowak,et al.  An EM algorithm for wavelet-based image restoration , 2003, IEEE Trans. Image Process..

[9]  Minh N. Do,et al.  Fast Global Image Smoothing Based on Weighted Least Squares , 2014, IEEE Transactions on Image Processing.

[10]  G. Sapiro,et al.  A collaborative framework for 3D alignment and classification of heterogeneous subvolumes in cryo-electron tomography. , 2013, Journal of structural biology.

[11]  Ming-Yu Liu,et al.  Joint Geodesic Upsampling of Depth Images , 2013, 2013 IEEE Conference on Computer Vision and Pattern Recognition.

[12]  Stephen P. Boyd,et al.  Proximal Algorithms , 2013, Found. Trends Optim..

[13]  G. Aubert,et al.  Image decomposition: application to textured images and SAR images , 2003 .

[14]  Horst Bischof,et al.  Image Guided Depth Upsampling Using Anisotropic Total Generalized Variation , 2013, 2013 IEEE International Conference on Computer Vision.

[15]  Stephen Lin,et al.  Data-driven depth map refinement via multi-scale sparse representation , 2015, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[16]  Zhenhua Guo,et al.  Color-Guided Depth Recovery via Joint Local Structural and Nonlocal Low-Rank Regularization , 2017, IEEE Transactions on Multimedia.

[17]  Jing-Yu Yang,et al.  Depth recovery via decomposition of polynomial and piece-wise constant signals , 2016, 2016 Visual Communications and Image Processing (VCIP).

[18]  D. Donoho,et al.  Simultaneous cartoon and texture image inpainting using morphological component analysis (MCA) , 2005 .

[19]  Stephen P. Boyd,et al.  Distributed Optimization and Statistical Learning via the Alternating Direction Method of Multipliers , 2011, Found. Trends Mach. Learn..

[20]  Robert D. Nowak,et al.  Majorization–Minimization Algorithms for Wavelet-Based Image Restoration , 2007, IEEE Transactions on Image Processing.

[21]  Qiang Wu,et al.  Explicit modeling on depth-color inconsistency for color-guided depth up-sampling , 2016, 2016 IEEE International Conference on Multimedia and Expo (ICME).

[22]  H. Vincent Poor,et al.  An Introduction to Signal Detection and Estimation , 1994, Springer Texts in Electrical Engineering.

[23]  Ivan W. Selesnick,et al.  Total Variation Denoising Via the Moreau Envelope , 2017, IEEE Signal Processing Letters.

[24]  Qiang Wu,et al.  Minimum spanning forest with embedded edge inconsistency measurement for color-guided depth map upsampling , 2017, 2017 IEEE International Conference on Multimedia and Expo (ICME).

[25]  Douglas L. Jones,et al.  Real-valued fast Fourier transform algorithms , 1987, IEEE Trans. Acoust. Speech Signal Process..

[26]  Qiang Wu,et al.  Robust Color Guided Depth Map Restoration , 2017, IEEE Transactions on Image Processing.

[27]  Björn E. Ottersten,et al.  Real-Time Distance-Dependent Mapping for a Hybrid ToF Multi-Camera Rig , 2012, IEEE Journal of Selected Topics in Signal Processing.

[28]  Shunsuke Ono,et al.  Cartoon-Texture Image Decomposition Using Blockwise Low-Rank Texture Characterization , 2014, IEEE Transactions on Image Processing.

[29]  Qingxiong Yang,et al.  A non-local cost aggregation method for stereo matching , 2012, 2012 IEEE Conference on Computer Vision and Pattern Recognition.

[30]  Dani Lischinski,et al.  Joint bilateral upsampling , 2007, SIGGRAPH 2007.

[31]  Jing-Yu Yang,et al.  Moiré pattern removal from texture images via low-rank and sparse matrix decomposition , 2015, 2015 Visual Communications and Image Processing (VCIP).

[32]  Heinz H. Bauschke,et al.  Convex Analysis and Monotone Operator Theory in Hilbert Spaces , 2011, CMS Books in Mathematics.

[33]  Michael Elad,et al.  Submitted to Ieee Transactions on Image Processing Image Decomposition via the Combination of Sparse Representations and a Variational Approach , 2022 .

[34]  Qiang Wu,et al.  Minimum Spanning Forest With Embedded Edge Inconsistency Measurement Model for Guided Depth Map Enhancement , 2018, IEEE Transactions on Image Processing.

[35]  Yao Wang,et al.  Color-Guided Depth Recovery From RGB-D Data Using an Adaptive Autoregressive Model , 2014, IEEE Transactions on Image Processing.

[36]  Lin Xie,et al.  Improved spatial pyramid matching for scene recognition , 2018, Pattern Recognit..

[37]  Patrick L. Combettes,et al.  Proximal Splitting Methods in Signal Processing , 2009, Fixed-Point Algorithms for Inverse Problems in Science and Engineering.

[38]  Minh N. Do,et al.  Fast Guided Global Interpolation for Depth and Motion , 2016, ECCV.

[39]  A. K. Riemens,et al.  Multistep joint bilateral depth upsampling , 2009, Electronic Imaging.

[40]  Björn E. Ottersten,et al.  Unified multi-lateral filter for real-time depth map enhancement , 2015, Image Vis. Comput..

[41]  Sebastian Thrun,et al.  Upsampling range data in dynamic environments , 2010, 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[42]  Michael S. Brown,et al.  High-Quality Depth Map Upsampling and Completion for RGB-D Cameras , 2014, IEEE Transactions on Image Processing.

[43]  Minh N. Do,et al.  Cross-based local multipoint filtering , 2012, 2012 IEEE Conference on Computer Vision and Pattern Recognition.

[44]  Ruigang Yang,et al.  Spatial-Depth Super Resolution for Range Images , 2007, 2007 IEEE Conference on Computer Vision and Pattern Recognition.

[45]  Jian Sun,et al.  Guided Image Filtering , 2010, ECCV.

[46]  Nirmal K. Bose,et al.  Superresolution and noise filtering using moving least squares , 2006, IEEE Transactions on Image Processing.

[47]  Jonathan T. Barron,et al.  The Fast Bilateral Solver , 2015, ECCV.