A Smart Post-Rectification Algorithm Based on an ANN Considering Reflectivity and Distance for Indoor Scenario Reconstruction

Indoor scene reconstruction is important for robot positioning and navigation in scenario reconstruction, especially in constructing a semantic map. In previous research, RGB-D cameras have been utilized to obtain a semantic map. However, because of indoor objects and depth sensors, the accuracy and precision of the depth values could be improved, which is a key factor in reconstructing indoor scenarios. Moreover, there is a relationship between reflectivity and depth accuracy. Therefore, to obtain depth information that is better than that obtained in our previous research, we present a smart post-rectification algorithm based on an artificial neural network (ANN). The algorithm improves the accuracy and precision of depth values by simultaneously considering reflectivity, distances, and different mechanisms of measuring depth. First, we analyze the RGB-D cameras’ characteristics, including the pinhole camera model, lens distortions, and the types of error factors due to the types of RGB-D cameras used. Then, this paper proposes a smart post-rectification algorithm for depth images based on an ANN considering the depth error caused by reflectivity, the distance-related depth error, and different mechanisms for measuring depth. Finally, we perform experiments to evaluate the accuracy and precision of the proposed post-rectification approach by using different types of depth sensors. To evaluate the performance of our proposed algorithm, the proposed approach is applied to RGB-D SLAM, which is tested in different indoor environments. The experimental results show that applying our post-rectification algorithm to indoor scenario reconstruction can result in more accurate and more detailed 3-D reconstruction of objects than other state-of-the-art methods, highlighting the robustness and efficiency of our proposed algorithm.

[1]  Matteo Munaro,et al.  Performance evaluation of the 1st and 2nd generation Kinect for multimedia applications , 2015, 2015 IEEE International Conference on Multimedia and Expo (ICME).

[2]  Diego Gonzalez-Aguilera,et al.  Low-Cost Reflectance-Based Method for the Radiometric Calibration of Kinect 2 , 2016, IEEE Sensors Journal.

[3]  Zhuowen Tu,et al.  Robust $L_{2}E$ Estimation of Transformation for Non-Rigid Registration , 2015, IEEE Transactions on Signal Processing.

[4]  Li Sun,et al.  A fully end-to-end deep learning approach for real-time simultaneous 3D reconstruction and material recognition , 2017, 2017 18th International Conference on Advanced Robotics (ICAR).

[5]  Pablo J. Zarco-Tejada,et al.  Tree height quantification using very high resolution imagery acquired from an unmanned aerial vehicle (UAV) and automatic 3D photo-reconstruction methods , 2014 .

[6]  Andreas Kolb,et al.  Kinect range sensing: Structured-light versus Time-of-Flight Kinect , 2015, Comput. Vis. Image Underst..

[7]  Thomas Butkiewicz Low-cost coastal mapping using Kinect v2 time-of-flight cameras , 2014, 2014 Oceans - St. John's.

[8]  Yao Wang,et al.  Color-Guided Depth Recovery From RGB-D Data Using an Adaptive Autoregressive Model , 2014, IEEE Transactions on Image Processing.

[9]  Klaus-Dieter Kuhnert,et al.  Fusion ofStereo-Camera andPMD-Camera DataforReal- TimeSuited Precise 3DEnvironment Reconstruction , 2006 .

[10]  Ludovico Minto,et al.  Operating Principles of Structured Light Depth Cameras , 2016 .

[11]  Stephen Lin,et al.  Shading-Based Shape Refinement of RGB-D Images , 2013, 2013 IEEE Conference on Computer Vision and Pattern Recognition.

[12]  Didier Stricker,et al.  Comparison of Kinect V1 and V2 Depth Images in Terms of Accuracy and Precision , 2016, ACCV Workshops.

[13]  Andreas Kolb,et al.  Calibration of the intensity-related distance error of the PMD TOF-camera , 2007, SPIE Optics East.

[14]  Reinhard Koch,et al.  Time-of-Flight sensor calibration for accurate range sensing , 2010, Comput. Vis. Image Underst..

[15]  Feng Zhao,et al.  A Graph-Based Approach for 3D Building Model Reconstruction from Airborne LiDAR Point Clouds , 2017, Remote. Sens..

[16]  Livio Pinto,et al.  Calibration of Kinect for Xbox One and Comparison between the Two Generations of Microsoft Sensors , 2015, Sensors.

[17]  Masatoshi Okutomi,et al.  Joint Estimation of Depth, Reflectance and Illumination for Depth Refinement , 2015, 2015 IEEE International Conference on Computer Vision Workshop (ICCVW).

[18]  In-So Kweon,et al.  High Quality Shape from a Single RGB-D Image under Uncalibrated Natural Illumination , 2013, 2013 IEEE International Conference on Computer Vision.

[19]  H. Macher,et al.  FIRST EXPERIENCES WITH KINECT V2 SENSOR FOR CLOSE RANGE 3D MODELLING , 2015 .

[20]  Wolfram Burgard,et al.  3-D Mapping With an RGB-D Camera , 2014, IEEE Transactions on Robotics.

[21]  Elise Lachat,et al.  Assessment and Calibration of a RGB-D Camera (Kinect v2 Sensor) Towards a Potential Use for Close-Range 3D Modeling , 2015, Remote. Sens..

[22]  Federico Tombari,et al.  Analysis and Evaluation Between the First and the Second Generation of RGB-D Sensors , 2015, IEEE Sensors Journal.

[23]  Qi Wu,et al.  A Post-Rectification Approach of Depth Images of Kinect v2 for 3D Reconstruction of Indoor Scenes , 2017, ISPRS Int. J. Geo Inf..

[24]  Albert S. Huang,et al.  Visual Odometry and Mapping for Autonomous Flight Using an RGB-D Camera , 2011, ISRR.

[25]  Stephen Lin,et al.  Data-driven depth map refinement via multi-scale sparse representation , 2015, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[26]  Andrew Zisserman,et al.  Very Deep Convolutional Networks for Large-Scale Image Recognition , 2014, ICLR.

[27]  Zhuowen Tu,et al.  Robust Point Matching via Vector Field Consensus , 2014, IEEE Transactions on Image Processing.

[28]  Stefan Leutenegger,et al.  SemanticFusion: Dense 3D semantic mapping with convolutional neural networks , 2016, 2017 IEEE International Conference on Robotics and Automation (ICRA).

[29]  Alberto Landi,et al.  Artificial Neural Networks for nonlinear regression and classification , 2010, 2010 10th International Conference on Intelligent Systems Design and Applications.

[30]  Roland Siegwart,et al.  Kinect v2 for mobile robot navigation: Evaluation and modeling , 2015, 2015 International Conference on Advanced Robotics (ICAR).

[31]  Rainer Koch,et al.  Multi-Robot Localization and Mapping Based on Signed Distance Functions , 2015, 2015 IEEE International Conference on Autonomous Robot Systems and Competitions.

[32]  Felix Järemo Lawin Depth Data Processing and 3D Reconstruction Using the Kinect v2 , 2015 .

[33]  Lembit Valgma 3D reconstruction using Kinect v2 camera , 2016 .

[34]  Pedro Arias,et al.  Metrological comparison between Kinect I and Kinect II sensors , 2015 .

[35]  Wenbing Tao,et al.  Progressive match expansion via coherent subspace constraint , 2016, Inf. Sci..

[36]  Bo Wang,et al.  3D reconstruction of building facade with fused data of terrestrial LiDAR data and optical image , 2016 .