Improving RGB Image Consistency for Depth-Camera based Reconstruction through Image Warping

Indoor reconstruction using depth camera algorithms (e.g., InfiniTAMv3) is becoming increasingly popular. Simple reconstruction methods solely use the frames of the depth camera, leaving any imagery from the adjunct RGB camera untouched. Recent approaches also incorporate color camera information to improve consistency. However, the results heavily depend on the accuracy of the rig calibration, which can strongly vary in quality. Unfortunately, any errors in the rig calibration result in apparent visual discrepancies when it comes to colorization of the 3D reconstruction. We propose an easy approach to fix this issue for the purpose of image-based rendering. We show that a relatively simple warping function can be calculated from a 3D checkerboard pattern for a rig with poor calibration between cameras. The warping is applied to the RGB images online during reconstruction, leading to a significantly improved visual result.

[1]  Friedrich Fraundorfer,et al.  Globally Consistent Dense Real-Time 3D Reconstruction from RGBD Data , 2018 .

[2]  Arjan Kuijper,et al.  [POSTER] Efficient Pose Selection for Interactive Camera Calibration , 2017, 2017 IEEE International Symposium on Mixed and Augmented Reality (ISMAR-Adjunct).

[3]  Jean-Yves Bouguet,et al.  Camera calibration toolbox for matlab , 2001 .

[4]  Janne Heikkilä,et al.  A four-step camera calibration procedure with implicit image correction , 1997, Proceedings of IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[5]  Weidong Geng,et al.  Accurate Intrinsic Calibration of Depth Camera with Cuboids , 2014, ECCV.

[6]  Jianping Li,et al.  Calibrate Multiple Consumer RGB-D Cameras for Low-Cost and Efficient 3D Indoor Mapping , 2018, Remote. Sens..

[7]  Joaquim Salvi,et al.  Pattern codification strategies in structured light systems , 2004, Pattern Recognit..

[8]  Remo Sala,et al.  A Survey on 3D Cameras: Metrological Comparison of Time-of-Flight, Structured-Light and Active Stereoscopy Technologies , 2018, SpringerBriefs in Computer Science.

[9]  HoraudRadu,et al.  An overview of depth cameras and range scanners based on time-of-flight technologies , 2016 .

[10]  Olaf Kähler,et al.  Very High Frame Rate Volumetric Integration of Depth Images on Mobile Devices , 2015, IEEE Transactions on Visualization and Computer Graphics.

[11]  Andrew W. Fitzgibbon,et al.  KinectFusion: Real-time dense surface mapping and tracking , 2011, 2011 10th IEEE International Symposium on Mixed and Augmented Reality.

[12]  Sebastian Thrun,et al.  Unsupervised Intrinsic Calibration of Depth Sensors via SLAM , 2013, Robotics: Science and Systems.

[13]  Horst Bischof,et al.  Learning Depth Calibration of Time-of-Flight Cameras , 2015, BMVC.

[14]  Ramesh Raskar,et al.  3D Depth Cameras in Vision: Benefits and Limitations of the Hardware , 2014 .

[15]  Dieter Schmalstieg,et al.  Real-Time View Planning for Unstructured Lumigraph Modeling , 2019, IEEE Transactions on Visualization and Computer Graphics.

[16]  Fabio Morbidi,et al.  Practical and accurate calibration of RGB-D cameras using spheres , 2015, Comput. Vis. Image Underst..

[17]  Bernd Fröhlich,et al.  Volumetric calibration and registration of multiple RGBD-sensors into a joint coordinate system , 2015, 2015 IEEE Symposium on 3D User Interfaces (3DUI).

[18]  George Drettakis,et al.  Scalable inside-out image-based rendering , 2016, ACM Trans. Graph..

[19]  Emanuele Menegatti,et al.  Robust Intrinsic and Extrinsic Calibration of RGB-D Cameras , 2017, IEEE Transactions on Robotics.

[20]  Wenbin Li,et al.  A Robust Calibration Method for Consumer Grade RGB-D Sensors for Precise Indoor Reconstruction , 2019, IEEE Access.

[21]  Juho Kannala,et al.  Joint Depth and Color Camera Calibration with Distortion Correction , 2012, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[22]  Wu Chen,et al.  A New Calibration Method for Commercial RGB-D Sensors , 2017, Sensors.

[23]  Zhengyou Zhang,et al.  A Flexible New Technique for Camera Calibration , 2000, IEEE Trans. Pattern Anal. Mach. Intell..

[24]  Tobias Höllerer,et al.  Real-Time Re-Textured Geometry Modeling Using Microsoft HoloLens , 2018, 2018 IEEE Conference on Virtual Reality and 3D User Interfaces (VR).

[25]  Andrew D. Payne,et al.  A 0.13 μm CMOS System-on-Chip for a 512 × 424 Time-of-Flight Image Sensor With Multi-Frequency Photo-Demodulation up to 130 MHz and 2 GS/s ADC , 2015, IEEE Journal of Solid-State Circuits.

[26]  Zhengyou Zhang,et al.  Calibration between depth and color sensors for commodity depth cameras , 2011, 2011 IEEE International Conference on Multimedia and Expo.

[27]  Radu Horaud,et al.  An overview of depth cameras and range scanners based on time-of-flight technologies , 2016, Machine Vision and Applications.

[28]  Robert Haschke,et al.  3D scene segmentation for autonomous robot grasping , 2012, 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[29]  Andreas Geiger,et al.  Automatic camera and range sensor calibration using a single shot , 2012, 2012 IEEE International Conference on Robotics and Automation.