Color-Guided Depth Map Super Resolution Using Convolutional Neural Network

With the development of 3-D applications, such as 3-D reconstruction and object recognition, accurate and high-quality depth map is urgently required. Recently, depth cameras have been affordable and widely used in daily life. However, the captured depth map always owns low resolution and poor quality, which limits its practical application. This paper proposes a color-guided depth map super resolution method using convolutional neural network. First, a dual-stream convolutional neural network, which integrates the color and depth information simultaneously, is proposed for depth map super resolution. Then, the optimized edge map generated by the high resolution color image and low resolution depth map is used as additional information to refine the object boundary in the depth map. Experimental results demonstrate the effectiveness of the proposed method compared with the state-of-the-art methods.

[1]  Xuelong Li,et al.  Learning Multilayer Channel Features for Pedestrian Detection , 2016, IEEE Transactions on Image Processing.

[2]  Ming-Yu Liu,et al.  Joint Geodesic Upsampling of Depth Images , 2013, 2013 IEEE Conference on Computer Vision and Pattern Recognition.

[3]  Dani Lischinski,et al.  Joint bilateral upsampling , 2007, SIGGRAPH 2007.

[4]  Hong Liu,et al.  Body structure based triplet Convolutional Neural Network for person re-identification , 2017, 2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[5]  Xiaoou Tang,et al.  Image Super-Resolution Using Deep Convolutional Networks , 2014, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[6]  Ruigang Yang,et al.  Spatial-Depth Super Resolution for Range Images , 2007, 2007 IEEE Conference on Computer Vision and Pattern Recognition.

[7]  Tony Jacob,et al.  Single Image Super-Resolution Using a Joint GMM Method , 2016, IEEE Transactions on Image Processing.

[8]  Kai-Lung Hua,et al.  Joint trilateral filtering for depth map super-resolution , 2013, 2013 Visual Communications and Image Processing (VCIP).

[9]  Christopher Zach,et al.  A dynamic programming approach for fast and robust object pose recognition from range images , 2015, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[10]  Qionghai Dai,et al.  Synthesis-guided depth super resolution , 2014, 2014 IEEE Visual Communications and Image Processing Conference.

[11]  Michael S. Brown,et al.  High quality depth map upsampling for 3D-TOF cameras , 2011, 2011 International Conference on Computer Vision.

[12]  Sebastian Thrun,et al.  An Application of Markov Random Fields to Range Sensing , 2005, NIPS.

[13]  Yehoshua Y. Zeevi,et al.  Image enhancement and denoising by complex diffusion processes , 2004, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[14]  Carsten Rother,et al.  Depth Super Resolution by Rigid Body Self-Similarity in 3D , 2013, 2013 IEEE Conference on Computer Vision and Pattern Recognition.

[15]  Jianjun Lei,et al.  Depth Map Super-Resolution Considering View Synthesis Quality , 2017, IEEE Transactions on Image Processing.

[16]  Rogério Schmidt Feris,et al.  Single depth image super resolution and denoising via coupled dictionary learning with local constraints and shock filtering , 2014, 2014 IEEE International Conference on Multimedia and Expo (ICME).

[17]  Hongbin Zha,et al.  Similarity-Aware Patchwork Assembly for Depth Image Super-resolution , 2014, 2014 IEEE Conference on Computer Vision and Pattern Recognition.

[18]  Weisi Lin,et al.  Saliency detection for stereoscopic images , 2013, 2013 Visual Communications and Image Processing (VCIP).

[19]  Ryo Furukawa,et al.  Dense 3D Reconstruction from High Frame-Rate Video Using a Static Grid Pattern , 2014, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[20]  Rogério Schmidt Feris,et al.  Joint Super Resolution and Denoising From a Single Depth Image , 2015, IEEE Transactions on Multimedia.

[21]  Lifeng Sun,et al.  Joint Example-Based Depth Map Super-Resolution , 2012, 2012 IEEE International Conference on Multimedia and Expo.

[22]  Rogério Schmidt Feris,et al.  Edge guided single depth image super resolution , 2014, 2014 IEEE International Conference on Image Processing (ICIP).

[23]  Xinpeng Zhang,et al.  Quaternion-type moments combining both color and depth information for RGB-D object recognition , 2016, 2016 23rd International Conference on Pattern Recognition (ICPR).

[24]  Gabriel J. Brostow,et al.  Patch Based Synthesis for Single Depth Image Super-Resolution , 2012, ECCV.

[25]  Xuelong Li,et al.  Classifying Discriminative Features for Blur Detection , 2016, IEEE Transactions on Cybernetics.

[26]  Fang Li,et al.  Image retrieval based on deep Convolutional Neural Networks and binary hashing learning , 2017, 2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[27]  Yoshua Bengio,et al.  Gradient-based learning applied to document recognition , 1998, Proc. IEEE.

[28]  Evgeniy Gabrilovich,et al.  A Review of Relational Machine Learning for Knowledge Graphs , 2015, Proceedings of the IEEE.

[29]  Andrew G. Howard,et al.  Some Improvements on Deep Convolutional Neural Network Based Image Classification , 2013, ICLR.

[30]  Honggang Zhang,et al.  Variational Bayesian Matrix Factorization for Bounded Support Data , 2015, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[31]  Horst Bischof,et al.  Image Guided Depth Upsampling Using Anisotropic Total Generalized Variation , 2013, 2013 IEEE International Conference on Computer Vision.

[32]  Zhen Yang,et al.  Decorrelation of Neutral Vector Variables: Theory and Applications , 2017, IEEE Transactions on Neural Networks and Learning Systems.

[33]  Seung-Won Jung,et al.  A Consensus-Driven Approach for Structure and Texture Aware Depth Map Upsampling , 2014, IEEE Transactions on Image Processing.