3D Mapping for Visualization of Rigid Structures: A Review and Comparative Study

In this review, we discuss state-of-the-art developments in 3D models for small and rigid structures. This includes the pros and cons of cutting-edge range cameras used as active 3D scanners, while also considering passive image reconstruction schemes by means of the well-known structure-from-motion (SfM) algorithms. Furthermore, we discuss the issue of how data fusion algorithms can be used to optimally fuse 2D contour information onto 3D models for several different applications. Considering the benefits of 3D range sensors, we also review current trends in optimum data fusion of point clouds from 3D range sensors. We present the benefits and the limitations of each algorithm against various design considerations. To highlight the pros and cons, we also perform a comparative study of the performance of a 3D range sensor, represented by an iPad structure sensor, with respect to the well-known SfM software packages, namely, Bundler, Microsoft PhotoSynth, Agisoft PhotoScan, and Smart3DCapture. Last, we highlight several research opportunities and potential research challenges associated with each technique.

[1]  Michael J. Black,et al.  Home 3D body scans from noisy image and range data , 2011, 2011 International Conference on Computer Vision.

[2]  Bijan Shirinzadeh,et al.  Experimental Analysis of Laser Interferometry-Based Robust Motion Tracking Control of a Flexure-Based Mechanism , 2013, IEEE Transactions on Automation Science and Engineering.

[3]  A. Ardeshir Goshtasby,et al.  Piecewise cubic mapping functions for image registration , 1987, Pattern Recognit..

[4]  Andrew W. Fitzgibbon,et al.  KinectFusion: real-time 3D reconstruction and interaction using a moving depth camera , 2011, UIST.

[5]  Reinhard Koch,et al.  Technical Foundation and Calibration Methods for Time-of-Flight Cameras , 2013, Time-of-Flight and Depth Imaging.

[6]  George Wolberg,et al.  Multiview Geometry for Texture Mapping 2D Images Onto 3D Range Data , 2006, 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'06).

[7]  Yongjian Sun,et al.  A method for customizing 3D virtual human body models based on Multi-class Support Vector Machine , 2011, 2011 Seventh International Conference on Natural Computation.

[8]  Andras Molnar,et al.  Unmanned aerial vehicle guidance methods for 3D aerial image reconstruction , 2013, 2013 IEEE 9th International Conference on Computational Cybernetics (ICCC).

[9]  Roger Y. Tsai,et al.  A versatile camera calibration technique for high-accuracy 3D machine vision metrology using off-the-shelf TV cameras and lenses , 1987, IEEE J. Robotics Autom..

[10]  Andrew Hogue,et al.  Reconstructing 3D Buildings from LIDAR Using Level Set Methods , 2013, 2013 International Conference on Computer and Robot Vision.

[11]  Suya You,et al.  3D Reconstruction of Urban Areas , 2011, 2011 International Conference on 3D Imaging, Modeling, Processing, Visualization and Transmission.

[12]  Michael Riis Andersen,et al.  Kinect Depth Sensor Evaluation for Computer Vision Applications , 2012 .

[13]  Luc Van Gool,et al.  Multibody Structure-from-Motion in Practice , 2010, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[14]  Pierre Payeur,et al.  Experimental characterization of two generations of Kinect's depth sensors , 2012, 2012 IEEE International Symposium on Robotic and Sensors Environments Proceedings.

[15]  Yuri Owechko,et al.  2D/3D Sensor Exploitation and Fusion for Enhanced Object Detection , 2014, 2014 IEEE Conference on Computer Vision and Pattern Recognition Workshops.

[16]  Matthijs C. Dorst.  Distinctive Image Features from Scale-Invariant Keypoints , 2011 .

[17]  Richard Green,et al.  Evaluation of real time stereo vision system using web cameras , 2010, 2010 25th International Conference of Image and Vision Computing New Zealand.

[18]  Jie Tian,et al.  3D Medical Imaging Computation and Analysis Platform , 2006, 2006 IEEE International Conference on Industrial Technology.

[19]  S. Robson,et al.  Straightforward reconstruction of 3D surfaces and topography with a camera: Accuracy and geoscience application , 2012 .

[20]  Luc Van Gool,et al.  SURF: Speeded Up Robust Features , 2006, ECCV.

[21]  M. Westoby,et al.  ‘Structure-from-Motion’ photogrammetry: A low-cost, effective tool for geoscience applications , 2012 .

[22]  Jun Wu,et al.  General Framework of Photo-Realistic 3D Visualization of Urban Building , 2009, 2009 Fifth International Conference on Image and Graphics.

[23]  Fei Wang,et al.  A mono-camera and scanning laser range finder based UAV indoor navigation system , 2013, 2013 International Conference on Unmanned Aircraft Systems (ICUAS).

[24]  A. Aydin Alatan,et al.  Fusing 2D and 3D clues for 3D tracking using visual and range data , 2013, Proceedings of the 16th International Conference on Information Fusion.

[25]  Nicholas M. DiFilippo,et al.  Characterization of Different Microsoft Kinect Sensor Models , 2015, IEEE Sensors Journal.

[26]  Gabriele Peters,et al.  The structure-from-motion reconstruction pipeline - a survey with focus on short image sequences , 2010, Kybernetika.

[27]  Long Quan,et al.  Structure from motion from three affine views , 2002, Object recognition supported by user interaction for service robots.

[28]  Paul Suetens,et al.  Comparative evaluation of multiresolution optimization strategies for multimodality image registration by maximization of mutual information , 1999, Medical Image Anal..

[29]  A. K. Wong,et al.  50th Anniversary Article: Seeing Stresses through the Thermoelastic Lens—A Retrospective and Prospective from an Australian Viewpoint , 2015 .

[30]  Daniel G. Aliaga,et al.  Pose-Free Structure From Motion Using Depth From Motion Constraints , 2011, IEEE Transactions on Image Processing.

[31]  Frank Dellaert,et al.  Structure from motion without correspondence , 2000, Proceedings IEEE Conference on Computer Vision and Pattern Recognition. CVPR 2000 (Cat. No.PR00662).

[32]  Jiang Yu Zheng,et al.  Virtual Recovery and Exhibition of Heritage , 2000, IEEE Multim..

[33]  Daniel Cremers,et al.  LSD-SLAM: Large-Scale Direct Monocular SLAM , 2014, ECCV.

[34]  Andrew W. Fitzgibbon,et al.  KinectFusion: Real-time dense surface mapping and tracking , 2011, 2011 10th IEEE International Symposium on Mixed and Augmented Reality.

[35]  Fabio Remondino,et al.  Image‐based 3D Modelling: A Review , 2006 .

[36]  Joachim Hertzberg,et al.  An autonomous mobile robot with a 3D laser range finder for 3D exploration and digitalization of indoor environments , 2003, Robotics Auton. Syst..

[37]  Jianwei Zhang,et al.  Adaptive sensor-fusion of depth and color information for cognitive robotics , 2011, 2011 IEEE International Conference on Robotics and Biomimetics.

[38]  Angelo Di Iorio,et al.  Research Articles in Simplified HTML: a Web-first format for HTML-based scholarly articles , 2017, PeerJ Comput. Sci..

[39]  Imran Khan.  Non-Rigid Structure-From-Motion With Uniqueness Constraint and Low Rank Matrix Fitting Factorization , 2014, IEEE Transactions on Multimedia.

[40]  Gamini Dissanayake,et al.  A robust RGB-D SLAM algorithm , 2012, 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[41]  Sebastian Thrun,et al.  SCAPE: shape completion and animation of people , 2005, SIGGRAPH 2005.

[42]  R. Siegwart,et al.  Towards autonomous mine inspection , 2014, Proceedings of the 2014 3rd International Conference on Applied Robotics for the Power Industry.

[43]  Suk-kyu Ryu,et al.  Thermal Stresses Analysis of 3‐D Interconnect , 2009 .

[44]  Sergej Fatikow,et al.  Laser measuring system for a flexible microrobot-based micromanipulation station , 2000, Proceedings. 2000 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS 2000) (Cat. No.00CH37113).

[45]  Robert C. Bolles,et al.  Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography , 1981, CACM.

[46]  David G. Lowe,et al.  Object recognition from local scale-invariant features , 1999, Proceedings of the Seventh IEEE International Conference on Computer Vision.

[47]  Takeo Kanade,et al.  Shape-from-silhouette of articulated objects and its use for human body kinematics estimation and motion capture , 2003, 2003 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2003. Proceedings..

[48]  Stefano Mattoccia,et al.  A compact, lightweight and energy efficient system for autonomous navigation based on 3D vision , 2014, 2014 IEEE/ASME 10th International Conference on Mechatronic and Embedded Systems and Applications (MESA).

[49]  Bernhard P. Wrobel,et al.  Multiple View Geometry in Computer Vision , 2001 .

[50]  Wing Au,et al.  3D scene modeling using sensor fusion with laser range finder and image sensor , 2005, 34th Applied Imagery and Pattern Recognition Workshop (AIPR'05).

[51]  Franz Leberl,et al.  3D roof details by 3D aerial vision , 2011, 2011 IEEE International Conference on Computer Vision Workshops (ICCV Workshops).

[52]  Naoyuki Kubota,et al.  Robot perception of unexpected objects based on human visual structure using a 3D range camera , 2011, 2011 IEEE International Conference on Systems, Man, and Cybernetics.

[53]  Rama Chellappa,et al.  Structure from Motion Using Sequential Monte Carlo Methods , 2001, Proceedings Eighth IEEE International Conference on Computer Vision. ICCV 2001.

[54]  Azriel Rosenfeld,et al.  Optimal edge-based shape detection , 2002, IEEE Trans. Image Process..

[55]  Fabio Remondino,et al.  Image-based Automated Reconstruction of the Great Buddha of Bamiyan, Afghanistan , 2003, 2003 Conference on Computer Vision and Pattern Recognition Workshop.

[56]  Achim J. Lilienthal,et al.  Comparative evaluation of range sensor accuracy for indoor mobile robotics and automated logistics applications , 2013, Robotics Auton. Syst..

[57]  Richard I. Hartley,et al.  In Defense of the Eight-Point Algorithm , 1997, IEEE Trans. Pattern Anal. Mach. Intell..

[58]  Baba C. Vemuri,et al.  Non-Rigid Multi-Modal Image Registration Using Cross-Cumulative Residual Entropy , 2007, International Journal of Computer Vision.

[59]  Miska M. Hannuksela,et al.  Joint de-noising and fusion of 2D video and depth map sequences sensed by low-powered tof range sensor , 2013, 2013 IEEE International Conference on Multimedia and Expo Workshops (ICMEW).

[60]  Pascal Vasseur,et al.  2D-3D camera fusion for visual odometry in outdoor environments , 2014, 2014 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[61]  Khaled Nassar,et al.  Structure-From-Motion for Earthwork Planning , 2011 .

[62]  Peter Cho,et al.  3D Organization of 2D Urban Imagery , 2007, 36th Applied Imagery Pattern Recognition Workshop (aipr 2007).

[63]  R. Badlishah Ahmad,et al.  A Comparison between Using SIFT and SURF for Characteristic Region Based Image Steganography , 2012 .

[64]  Stephan Wong,et al.  Hand Segmentation by Fusing 2D and 3D Data , 2010, 2010 Second International Conference on Computer Modeling and Simulation.

[65]  Yaser Sheikh,et al.  Structure from motion blur in low light , 2011, CVPR 2011.

[66]  Wolfram Burgard,et al.  OctoMap : A Probabilistic , Flexible , and Compact 3 D Map Representation for Robotic Systems , 2010 .

[67]  Silvio Savarese,et al.  Semantic Structure from Motion: A Novel Framework for Joint Object Recognition and 3D Reconstruction , 2011, Theoretical Foundations of Computer Vision.

[68]  Aníbal Ollero,et al.  A cooperative perception system for multiple UAVs: Application to automatic detection of forest fires , 2006, J. Field Robotics.

[69]  Partha Pratim Das,et al.  Characterizations of Noise in Kinect Depth Images: A Review , 2014, IEEE Sensors Journal.

[70]  Silvio Savarese,et al.  Semantic structure from motion , 2011, CVPR 2011.

[71]  Paulo Dias,et al.  Registration and fusion of intensity and range data for 3D modelling of real world scenes , 2003, Fourth International Conference on 3-D Digital Imaging and Modeling, 2003. 3DIM 2003. Proceedings..

[72]  D. Delparte,et al.  Distributed under Creative Commons Cc-by 4.0 Integrating Structure-from-motion Photogrammetry with Geospatial Software as a Novel Technique for Quantifying 3d Ecological Characteristics of Coral Reefs , 2022 .

[73]  William E. Carter,et al.  Archaeological prospection of north Eastern Honduras with airborne mapping LiDAR , 2014, 2014 IEEE Geoscience and Remote Sensing Symposium.

[74]  Heinz Hügli,et al.  Fusion of Time of Flight Camera Point Clouds , 2008 .

[75]  Carl Salvaggio,et al.  Fusion of Terrestrial LiDAR Point Clouds with Color Imagery Senior Project : Colin , 2013 .

[76]  Richard Szeliski,et al.  Modeling the World from Internet Photo Collections , 2008, International Journal of Computer Vision.

[77]  Tom Duckett,et al.  3D Modeling of Indoor Environments for a Robotic Security Guard , 2005, 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05) - Workshops.

[78]  A. Ardeshir Goshtasby,et al.  Piecewise linear mapping functions for image registration , 1986, Pattern Recognit..

[79]  Mark R. Pickering,et al.  A new multi-modal similarity measure for fast gradient-based 2D-3D image registration , 2009, 2009 Annual International Conference of the IEEE Engineering in Medicine and Biology Society.

[80]  Karl C. Walli,et al.  Error characterization of flight trajectories reconstructed using Structure from Motion , 2013, 2013 IEEE Applied Imagery Pattern Recognition Workshop (AIPR).

[81]  D. Rubin,et al.  Maximum likelihood from incomplete data via the EM - algorithm plus discussions on the paper , 1977 .

[82]  Paolo Favaro,et al.  Shape from Focus and Defocus: Convexity, Quasiconvexity and Defocus-Invariant Textures , 2007, 2007 IEEE 11th International Conference on Computer Vision.

[83]  John Oliensis.  Exact Two-Image Structure from Motion , 2002, IEEE Trans. Pattern Anal. Mach. Intell..

[84]  Nassir Navab,et al.  Recovering projection geometry: how a cheap camera can outperform an expensive stereo system , 2000, Proceedings IEEE Conference on Computer Vision and Pattern Recognition. CVPR 2000 (Cat. No.PR00662).

[85]  David A. Forsyth,et al.  The Joy of Sampling , 2004, International Journal of Computer Vision.

[86]  Aboelmagd Noureldin,et al.  Integrated Indoor Navigation System for Ground Vehicles With Automatic 3-D Alignment and Position Initialization , 2015, IEEE Transactions on Vehicular Technology.

[87]  Steven M. Seitz,et al.  Photo tourism: exploring photo collections in 3D , 2006, ACM Trans. Graph..

[88]  Sunil Arya,et al.  An optimal algorithm for approximate nearest neighbor searching fixed dimensions , 1998, JACM.

[89]  Maxime Lhuillier.  Incremental Fusion of Structure-from-Motion and GPS Using Constrained Bundle Adjustments , 2012, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[90]  Jinhui Lan,et al.  New approach of imagery generation and target recognition based on 3D LIDAR data , 2009, 2009 9th International Conference on Electronic Measurement & Instruments.

[91]  S. Oprisescu,et al.  Automatic 2D/3D ToF image fusion using the intrinsic dimension , 2012, 2012 9th International Conference on Communications (COMM).

[92]  Ramin Samadani,et al.  Stereo matching based on image triangulation for view synthesis , 2012, 2012 19th IEEE International Conference on Image Processing.

[93]  Michael J. Cree,et al.  Advantages of 3D Time-of-Flight Range Imaging Cameras in Machine Vision Applications , 2009 .

[94]  Anh Vu,et al.  Targeted trajectories optimization from 2D LIDAR measurements , 2013, 2013 IEEE Intelligent Vehicles Symposium (IV).

[95]  Hideo Saito,et al.  Shape from Shading Using Genetic Algorithm , 1994 .

[96]  Warren E. Dixon,et al.  Single Camera Structure and Motion , 2012, IEEE Transactions on Automatic Control.

[97]  Amelia Wong Azman,et al.  Indoor UAV Positioning Using Stereo Vision Sensor , 2012 .

[98]  Juha Hyyppä,et al.  TerraSAR-X Stereo Radargrammetry and Airborne Scanning LiDAR Height Metrics in Imputation of Forest Aboveground Biomass and Stem Volume , 2014, IEEE Transactions on Geoscience and Remote Sensing.

[99]  Ming Wang,et al.  Indoor environment applications for mobile robots using Kinect2.0 , 2014, Proceeding of the 11th World Congress on Intelligent Control and Automation.

[100]  Jan Flusser,et al.  Image registration methods: a survey , 2003, Image Vis. Comput..

[101]  Nico Blodow,et al.  Towards 3D Point cloud based object maps for household environments , 2008, Robotics Auton. Syst..

[102]  Terrance E. Boult,et al.  The integration of information from stereo and multiple shape-from-texture cues , 1988, Proceedings CVPR '88: The Computer Society Conference on Computer Vision and Pattern Recognition.

[103]  Andreas Geiger,et al.  Vision meets robotics: The KITTI dataset , 2013, Int. J. Robotics Res..

[104]  Qionghai Dai,et al.  Extracting Depth and Radiance From a Defocused Video Pair , 2015, IEEE Transactions on Circuits and Systems for Video Technology.

[105]  Tomás Pajdla,et al.  3D with Kinect , 2011, 2011 IEEE International Conference on Computer Vision Workshops (ICCV Workshops).

[106]  Berthold K. P. Horn,et al.  Pattern design and imaging methods in 3-D coded aperture techniques , 1998, 1998 IEEE Nuclear Science Symposium Conference Record. 1998 IEEE Nuclear Science Symposium and Medical Imaging Conference (Cat. No.98CH36255).

[107]  Dongxiao Li,et al.  A Depth Extraction Method Based on Motion and Geometry for 2D to 3D Conversion , 2009, 2009 Third International Symposium on Intelligent Information Technology Application.

[108]  Mark Hardy,et al.  Autonomous inspection using an underwater 3D LiDAR , 2013, 2013 OCEANS - San Diego.

[109]  Reinhard Koch,et al.  Time-of-Flight Sensors in Computer Graphics , 2009, Eurographics.

[110]  Ashutosh Saxena,et al.  3-D Depth Reconstruction from a Single Still Image , 2007, International Journal of Computer Vision.

[111]  Yasemin Yardimci,et al.  Effective Structure-from-Motion for Hybrid Camera Systems , 2010, 2010 20th International Conference on Pattern Recognition.