Drone-based AI and 3D Reconstruction for Digital Twin Augmentation

Digital Twin is an emerging technology at the forefront of Industry 4.0, with the ultimate goal of combining the physical space and the virtual space. To date, the Digital Twin concept has been applied in many engineering fields, providing useful insights in the areas of engineering design, manufacturing, automation, and construction industry. While the nexus of various technologies opens up new opportunities with Digital Twin, the technology requires a framework to integrate the different technologies, such as the Building Information Model used in the Building and Construction industry. In this work, an Information Fusion framework is proposed to seamlessly fuse heterogeneous components in a Digital Twin framework from the variety of technologies involved. This study aims to augment Digital Twin in buildings with the use of AI and 3D reconstruction empowered by unmanned aviation vehicles. We proposed a drone-based Digital Twin augmentation framework with reusable and customisable components. A proof of concept is also developed, and extensive evaluation is conducted for 3D reconstruction and applications of AI for defect detection.

[1]  Robert J. Thomas,et al.  SDNET2018: An annotated image dataset for non-contact concrete crack detection using deep convolutional neural networks , 2018, Data in brief.

[2]  Robert C. Bolles,et al.  Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography , 1981, CACM.

[3]  Ilya Afanasyev,et al.  Comparison of Various SLAM Systems for Mobile Robot in an Indoor Environment , 2018, 2018 International Conference on Intelligent Systems (IS).

[4]  Maurice Murphy,et al.  Integration of Historic Building Information Modeling (HBIM) and 3D GIS for recording and managing cultural heritage sites , 2012, 2012 18th International Conference on Virtual Systems and Multimedia.

[5]  Azad M. Madni,et al.  Leveraging Digital Twin Technology in Model-Based Systems Engineering , 2019, Syst..

[6]  He Zhang,et al.  Digital Twin in Industry: State-of-the-Art , 2019, IEEE Transactions on Industrial Informatics.

[7]  Cettina Santagati,et al.  IMAGE-BASED MODELING TECHNIQUES FOR ARCHITECTURAL HERITAGE 3D DIGITALIZATION: LIMITS AND POTENTIALITIES , 2013 .

[8]  Charles M. Eastman,et al.  Building Information Modeling (BIM) and Safety: Automatic Safety Checking of Construction Models and Schedules , 2013 .

[9]  Wolfgang Hess,et al.  Real-time loop closure in 2D LIDAR SLAM , 2016, 2016 IEEE International Conference on Robotics and Automation (ICRA).

[10]  Sisi Zlatanova,et al.  INTEGRATION OF 3D OBJECTS AND TERRAIN FOR 3D MODELLING SUPPORTING THE DIGITAL TWIN , 2019, ISPRS Annals of the Photogrammetry, Remote Sensing and Spatial Information Sciences.

[11]  Ji Zhang,et al.  LOAM: Lidar Odometry and Mapping in Real-time , 2014, Robotics: Science and Systems.

[12]  F. Poux,et al.  AUTOMATIC 3D BUILDINGS COMPACT RECONSTRUCTION FROM LIDAR POINT CLOUDS , 2020, The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences.

[13]  Airborne LiDAR for DEM generation: some critical issues , 2008 .

[14]  Gabriel Dulac-Arnold,et al.  L2RPN: Learning to Run a Power Network in a Sustainable World NeurIPS2020 challenge design , 2020 .

[15]  Sankaran Mahadevan,et al.  A dynamic Bayesian network approach for digital twin , 2017 .

[16]  T. Fujimoto,et al.  The Effect of “ Front-Loading ” Problem-Solving on Product Development Performance , 2000 .

[17]  Matthew Louis Mauriello,et al.  Towards automated thermal profiling of buildings at scale using unmanned aerial vehicles and 3D-reconstruction , 2014, UbiComp Adjunct.

[18]  Siddhartha S. Srinivasa,et al.  Large-scale, real-time 3D scene reconstruction on a mobile device , 2017, Auton. Robots.

[19]  H. Friedrich,et al.  Large wood (LW) 3D accumulation mapping and assessment using structure from Motion photogrammetry in the laboratory , 2020 .

[20]  Dieter Fox,et al.  RGB-D Mapping: Using Depth Cameras for Dense 3D Modeling of Indoor Environments , 2010, ISER.

[21]  Mikael Hedlind,et al.  Digital Twin of a Cutting Tool , 2018 .

[22]  Stefan Kohlbrecher,et al.  A flexible and scalable SLAM system with full 3D motion estimation , 2011, 2011 IEEE International Symposium on Safety, Security, and Rescue Robotics.

[23]  Meng Zhang,et al.  Digital Twin Shop-Floor: A New Shop-Floor Paradigm Towards Smart Manufacturing , 2017, IEEE Access.

[24]  Xiaoye Liu,et al.  Airborne LiDAR for DEM generation: some critical issues , 2008 .

[25]  Michal Tolgyessy,et al.  3D map reconstruction with sensor kinect: Searching for solution applicable to small mobile robots , 2014, 2014 23rd International Conference on Robotics in Alpe-Adria-Danube Region (RAAD).

[26]  Vladlen Koltun,et al.  Robust reconstruction of indoor scenes , 2015, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[27]  Andrew Kusiak,et al.  Smart manufacturing must embrace big data , 2017, Nature.

[28]  Wei Wang,et al.  LIO-SAM: Tightly-coupled Lidar Inertial Odometry via Smoothing and Mapping , 2020, 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS).

[29]  Jack Chin Pang Cheng,et al.  BIMCloud: A Distributed Cloud-Based Social BIM Framework for Project Collaboration , 2014 .

[30]  Feng Zhao,et al.  A Graph-Based Approach for 3D Building Model Reconstruction from Airborne LiDAR Point Clouds , 2017, Remote. Sens..

[31]  LiDAR-Based 3D SLAM for Indoor Mapping , 2021, 2021 7th International Conference on Control, Automation and Robotics (ICCAR).

[32]  François Michaud,et al.  RTAB‐Map as an open‐source lidar and visual simultaneous localization and mapping library for large‐scale and long‐term online operation , 2018, J. Field Robotics.

[33]  Andrew Y. C. Nee,et al.  Digital twin-driven product design framework , 2019, Int. J. Prod. Res..

[34]  Jie Shan,et al.  Segmentation and Reconstruction of Polyhedral Building Roofs From Aerial Lidar Point Clouds , 2010, IEEE Transactions on Geoscience and Remote Sensing.

[35]  Zhong Fan,et al.  Digital Twin: Enabling Technologies, Challenges and Open Research , 2020, IEEE Access.

[36]  Chang-Su Shim,et al.  Development of a bridge maintenance system for prestressed concrete bridges using 3D digital twin model , 2019, Structure and Infrastructure Engineering.

[37]  Geoff Zeiss,et al.  Building Information Modeling , 2021, Wendehorst Bautechnische Zahlentafeln.

[38]  Wolfram Burgard,et al.  Improved Techniques for Grid Mapping With Rao-Blackwellized Particle Filters , 2007, IEEE Transactions on Robotics.

[39]  Luc Van Gool,et al.  Speeded-Up Robust Features (SURF) , 2008, Comput. Vis. Image Underst..

[40]  Eddy Krygiel,et al.  Green BIM: Successful Sustainable Design with Building Information Modeling , 2008 .

[41]  Alex Alcocer,et al.  Towards Automated 3D reconstruction in SME factories and Digital Twin Model generation , 2018, 2018 18th International Conference on Control, Automation and Systems (ICCAS).

[42]  Stephen Emmitt,et al.  Social BIM: Co-creation with shared situational awareness , 2015, J. Inf. Technol. Constr..

[43]  Piyush Maheshwari,et al.  The Convergence of Digital Twin, IoT, and Machine Learning: Transforming Data into Action , 2019, Internet of Things.

[44]  Jan-Michael Frahm,et al.  Structure-from-Motion Revisited , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[45]  George Chryssolouris,et al.  Digital twin-driven supervised machine learning for the development of artificial intelligence applications in manufacturing , 2020, Int. J. Comput. Integr. Manuf..

[46]  Fei Tao,et al.  Digital twin-driven product design, manufacturing and service with big data , 2017, The International Journal of Advanced Manufacturing Technology.

[47]  Cheng Zhang,et al.  Disaster City Digital Twin: A vision for integrating artificial and human intelligence for disaster management , 2021, Int. J. Inf. Manag..

[48]  Andrew Y. C. Nee,et al.  Enabling technologies and tools for digital twin , 2019 .

[49]  Edward H. Glaessgen,et al.  The Digital Twin Paradigm for Future NASA and U.S. Air Force Vehicles , 2012 .