Spatial Hierarchy Aware Residual Pyramid Network for Time-of-Flight Depth Denoising

Our proposed Spatial Hierarchy Aware Residual Pyramid Network (SHARPNet) consists of three parts: a Residual Regression Module as the backbone for multi-scale feature extraction, a Residual Fusion Module and a Depth Refinement Module to optimize the performance. The details of SHARP-Net are shown in Table 1. The (×) represents the upsample operation based on bicubic interpolation. For example, (×2) means that interpolating the input image to twice over its original size. The ‘all ©’ represents concating the upsample of output residuals of all the residual regression blocks. The ⊕ and the © respectively represent the addition operation and the concatenation operation.

[1]  Mark Meyer,et al.  Kernel-predicting convolutional networks for denoising Monte Carlo renderings , 2017, ACM Trans. Graph..

[2]  Leonidas J. Guibas,et al.  StructureNet , 2019, ACM Trans. Graph..

[3]  Angel X. Chang,et al.  Hierarchy Denoising Recursive Autoencoders for 3D Scene Layout Prediction , 2019, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[4]  Christoph S. Garbe,et al.  Denoising Time-Of-Flight Data with Adaptive Total Variation , 2011, ISVC.

[5]  Pat O'Connor,et al.  7.6 A 512×424 CMOS 3D Time-of-Flight image sensor with multi-frequency photo-demodulation up to 130MHz and 2GS/s ADC , 2014, 2014 IEEE International Solid-State Circuits Conference Digest of Technical Papers (ISSCC).

[6]  Tao Mei,et al.  Hierarchy Parsing for Image Captioning , 2019, 2019 IEEE/CVF International Conference on Computer Vision (ICCV).

[7]  Stefan Fuchs,et al.  Multipath Interference Compensation in Time-of-Flight Camera Images , 2010, 2010 20th International Conference on Pattern Recognition.

[8]  Yuan Yu,et al.  TensorFlow: A system for large-scale machine learning , 2016, OSDI.

[9]  Manuel Mazo,et al.  Modeling and correction of multipath interference in time of flight cameras , 2014, Image Vis. Comput..

[10]  MOHIT GUPTA,et al.  Phasor Imaging , 2015, ACM Trans. Graph..

[11]  Song Zhang,et al.  High-speed 3D shape measurement with structured light methods: A review , 2018, Optics and Lasers in Engineering.

[12]  Gordon Wetzstein,et al.  Deep End-to-End Time-of-Flight Imaging , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[13]  In-So Kweon,et al.  Time-of-Flight Sensor Calibration for a Color and Depth Camera Pair , 2015, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[14]  Didier Stricker,et al.  CoRBS: Comprehensive RGB-D benchmark for SLAM using Kinect v2 , 2016, 2016 IEEE Winter Conference on Applications of Computer Vision (WACV).

[15]  Piergiorgio Sartor,et al.  Unsupervised Domain Adaptation for ToF Data Denoising With Adversarial Learning , 2019, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[16]  Xiang Cao,et al.  Joint residual pyramid for joint image super-resolution , 2019, J. Vis. Commun. Image Represent..

[17]  Di Qiu,et al.  Deep End-to-End Alignment and Refinement for Time-of-Flight RGB-D Module , 2019, 2019 IEEE/CVF International Conference on Computer Vision (ICCV).

[18]  Joachim Denzler,et al.  Hierarchy-Based Image Embeddings for Semantic Image Retrieval , 2018, 2019 IEEE Winter Conference on Applications of Computer Vision (WACV).

[19]  Jechang Jeong,et al.  Densely Connected Hierarchical Network for Image Denoising , 2019, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[20]  Dong Liu,et al.  Zero-Shot Depth Estimation From Light Field Using A Convolutional Neural Network , 2020, IEEE Transactions on Computational Imaging.

[21]  Mirko Schmidt,et al.  SRA: Fast Removal of General Multipath for ToF Sensors , 2014, ECCV.

[22]  Olaf Hellwich,et al.  Compensation for Multipath in ToF Camera Measurements Supported by Photometric Calibration and Environment Integration , 2013, ICVS.

[23]  Xuejin Chen,et al.  Structure-Aware Residual Pyramid Network for Monocular Depth Estimation , 2019, IJCAI.

[24]  Min H. Kim,et al.  DeepToF: off-the-shelf real-time correction of multipath interference in time-of-flight imaging , 2017, ACM Trans. Graph..

[25]  Jan Kautz,et al.  Tackling 3D ToF Artifacts Through Learning and the FLAT Dataset , 2018, ECCV.

[26]  Zhiwei Xiong,et al.  Fusion of Time-of-Flight and Phase Shifting for high-resolution and low-latency depth sensing , 2015, 2015 IEEE International Conference on Multimedia and Expo (ICME).

[27]  Xu Zhao,et al.  EdgeStereo: A Context Integrated Residual Pyramid Network for Stereo Matching , 2018, ACCV.

[28]  Diego Gutierrez,et al.  A framework for transient rendering , 2014, ACM Trans. Graph..

[29]  Rui Yan,et al.  An Event-based Hierarchy Model for Object Recognition , 2019, 2019 IEEE Symposium Series on Computational Intelligence (SSCI).

[30]  Lianfa Bai,et al.  Residual Pyramid Learning for Single-Shot Semantic Segmentation , 2020, IEEE Transactions on Intelligent Transportation Systems.

[31]  R. Raskar,et al.  Resolving Multipath Interference in Kinect: An Inverse Problem Approach , 2014, IEEE Sensors Journal.