Depth superresolution using motion adaptive regularization

Spatial resolution of depth sensors is often significantly lower compared to that of conventional optical cameras. Recent work has explored the idea of improving the resolution of depth using higher resolution intensity as a side information. In this paper, we demonstrate that further incorporating temporal information in videos can significantly improve the results. In particular, we propose a novel approach that improves depth resolution, exploiting the space-time redundancy in the depth and intensity using motion-adaptive low-rank regularization. Experiments confirm that the proposed approach substantially improves the quality of the estimated high-resolution depth. Our approach can be a first component in systems using vision techniques that rely on high resolution depth information.

[1]  Jean-Michel Morel,et al.  Image Denoising Methods. A New Nonlocal Principle , 2010, SIAM Rev..

[2]  Sebastian Thrun,et al.  LidarBoost: Depth superresolution for ToF 3D shape scanning , 2009, CVPR.

[3]  S. Frick,et al.  Compressed Sensing , 2014, Computer Vision, A Reference Guide.

[4]  Emmanuel J. Candès,et al.  Robust uncertainty principles: exact signal reconstruction from highly incomplete frequency information , 2004, IEEE Transactions on Information Theory.

[5]  Sebastian Thrun,et al.  A Noise‐aware Filter for Real‐time Depth Upsampling , 2008 .

[6]  Michael Ying Yang,et al.  Joint Object Segmentation and Depth Upsampling , 2015, IEEE Signal Processing Letters.

[7]  Rick Chartrand,et al.  Nonconvex Splitting for Regularized Low-Rank + Sparse Decomposition , 2012, IEEE Transactions on Signal Processing.

[8]  Shree K. Nayar,et al.  Multiple view image denoising , 2009, 2009 IEEE Conference on Computer Vision and Pattern Recognition.

[9]  Ruigang Yang,et al.  Spatial-Depth Super Resolution for Range Images , 2007, 2007 IEEE Conference on Computer Vision and Pattern Recognition.

[10]  Mathews Jacob,et al.  A Fast Majorize–Minimize Algorithm for the Recovery of Sparse and Low-Rank Matrices , 2012, IEEE Transactions on Image Processing.

[11]  Timo Schairer,et al.  Fusion of range and color images for denoising and resolution enhancement with a non-local filter , 2010, Comput. Vis. Image Underst..

[12]  Xiaojin Gong,et al.  Guided Depth Upsampling via a Cosparse Analysis Model , 2014, 2014 IEEE Conference on Computer Vision and Pattern Recognition Workshops.

[13]  Andreas Geiger,et al.  Vision meets robotics: The KITTI dataset , 2013, Int. J. Robotics Res..

[14]  D K Smith,et al.  Numerical Optimization , 2001, J. Oper. Res. Soc..

[15]  Lifeng Sun,et al.  Joint Example-Based Depth Map Super-Resolution , 2012, 2012 IEEE International Conference on Multimedia and Expo.

[16]  Alessandro Foi,et al.  Image Denoising by Sparse 3-D Transform-Domain Collaborative Filtering , 2007, IEEE Transactions on Image Processing.

[17]  L. Rudin,et al.  Nonlinear total variation based noise removal algorithms , 1992 .

[18]  Ming-Yu Liu,et al.  Joint Geodesic Upsampling of Depth Images , 2013, 2013 IEEE Conference on Computer Vision and Pattern Recognition.

[19]  Feng Liu,et al.  Depth Enhancement via Low-Rank Matrix Completion , 2014, 2014 IEEE Conference on Computer Vision and Pattern Recognition.

[20]  Karen O. Egiazarian,et al.  BM3D Frames and Variational Image Deblurring , 2011, IEEE Transactions on Image Processing.

[21]  Aram Danielyan,et al.  Block-based Collaborative 3-D Transform Domain Modeling in Inverse Imaging , 2013 .

[22]  Sebastian Thrun,et al.  Upsampling range data in dynamic environments , 2010, 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[23]  Horst Bischof,et al.  Image Guided Depth Upsampling Using Anisotropic Total Generalized Variation , 2013, 2013 IEEE International Conference on Computer Vision.

[24]  Michael S. Brown,et al.  High quality depth map upsampling for 3D-TOF cameras , 2011, 2011 International Conference on Computer Vision.

[25]  Sebastian Thrun,et al.  An Application of Markov Random Fields to Range Sensing , 2005, NIPS.

[26]  Jian Sun,et al.  Guided Image Filtering , 2010, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[27]  Jong Chul Ye,et al.  Motion Adaptive Patch-Based Low-Rank Approach for Compressed Sensing Cardiac Cine MRI , 2014, IEEE Transactions on Medical Imaging.

[28]  Sebastian Thrun,et al.  LidarBoost: Depth superresolution for ToF 3D shape scanning , 2009, 2009 IEEE Conference on Computer Vision and Pattern Recognition.

[29]  Stephen P. Boyd,et al.  Distributed Optimization and Statistical Learning via the Alternating Direction Method of Multipliers , 2011, Found. Trends Mach. Learn..

[30]  Mathews Jacob,et al.  Nonlocal Regularization of Inverse Problems: A Unified Variational Framework , 2013, IEEE Transactions on Image Processing.

[31]  Dani Lischinski,et al.  Joint bilateral upsampling , 2007, ACM Trans. Graph..