Augmented reality system for facility management using image-based indoor localization

Abstract Image-based localization has provided opportunities for efficient facility management. Combined with augmented reality (AR), automated localization can offer visually assistive information in facility management. However, implementing an AR-based facility management system with image-based localization is difficult. Device-intensive methods or markers were prerequisites for facility information display. Localization accuracy and information readability and accessibility were some of the issues to be resolved for a successful representation of facility information. This paper presents an AR system for facility management using an image-based indoor localization method that estimates the user's indoor position and orientation by comparing the user's perspective to building information modeling (BIM) based on a deep learning computation. A graphics processing unit (GPU)-enabled server is used for the deep learning computation, and the resultant information is wirelessly transferred to the mobile AR device through transmission control protocol/Internet protocol (TCP/IP). Thereafter, spatial mapping visually fits the object of interest (e.g. pipes) onto the AR image using three-dimensional (3D) sensing capability of AR device. Experts evaluated that the proposed system has potential for improved facility management and identified future research direction, such as integrated information presentation and effective reflection of rehabilitation efforts on the drawings.

[1]  Mani Golparvar-Fard,et al.  Rapid Image-based Localization using Clustered 3D Point Cloud Models with Geo-Location Data for AEC/FM Mobile Augmented Reality Applications , 2014 .

[2]  Javier Irizarry,et al.  Investigating human and technological requirements for successful implementation of a BIM-based mobile augmented reality environment in facility management practices , 2016 .

[3]  Mani Golparvar-Fard,et al.  High-precision vision-based mobile augmented reality system for context-aware architectural, engineering, construction and facility management (AEC/FM) applications , 2013 .

[4]  Torsten Sattler,et al.  Image Retrieval for Image-Based Localization Revisited , 2012, BMVC.

[5]  Joaquim Salvi,et al.  Indoor SLAM using a range-augmented omnidirectional vision , 2011, 2011 IEEE International Conference on Computer Vision Workshops (ICCV Workshops).

[6]  Javier Irizarry,et al.  InfoSPOT: A mobile Augmented Reality method for accessing building information through a situation awareness approach , 2013 .

[7]  Kwang Gi Kim,et al.  Book Review: Deep Learning , 2016, Healthcare Informatics Research.

[8]  Yen-Pei Chen,et al.  Developing Mobile BIM/2D Barcode-Based Automated Facility Management System , 2014, TheScientificWorldJournal.

[9]  Ana Cristina Murillo,et al.  SURF features for efficient robot localization with omnidirectional images , 2007, Proceedings 2007 IEEE International Conference on Robotics and Automation.

[10]  Michael Suppa,et al.  Stereo-vision based obstacle mapping for indoor/outdoor SLAM , 2014, 2014 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[11]  Daniel Cremers,et al.  LSD-SLAM: Large-Scale Direct Monocular SLAM , 2014, ECCV.

[12]  Markus König,et al.  Combining visual natural markers and IMU for improved AR based indoor navigation , 2017, Adv. Eng. Informatics.

[13]  Lin Ma,et al.  Visual Map Construction Using RGB-D Sensors for Image-Based Localization in Indoor Environments , 2017, J. Sensors.

[14]  Hao Hong,et al.  BIM-Based Indoor Positioning Technology Using a Monocular Camera , 2017 .

[15]  Mark Ollila,et al.  UMAR: Ubiquitous Mobile Augmented Reality , 2004, MUM '04.

[16]  Lin Ma,et al.  Vision-based indoor localization approach based on SURF and landmark , 2016, 2016 International Wireless Communications and Mobile Computing Conference (IWCMC).

[17]  Kiyoharu Aizawa,et al.  Image-based indoor positioning system: fast image matching using omnidirectional panoramic images , 2010, MPVA '10.

[18]  Geoffrey E. Hinton,et al.  Deep Learning , 2015, Nature.

[19]  Khoa D. Phan,et al.  Indoor Slam Using an Omnidirectional Camera , 2013 .

[20]  Ömer Akin,et al.  Augmented reality-based computational fieldwork support for equipment operations and maintenance , 2011 .

[21]  Chun-Yi Su,et al.  RGB-D sensor-based visual SLAM for localization and navigation of indoor mobile robot , 2016, 2016 International Conference on Advanced Robotics and Mechatronics (ICARM).

[22]  Etienne Colle,et al.  A model-based method for indoor mobile robot localization using monocular vision and straight-line correspondences , 2005, Robotics Auton. Syst..

[23]  Guoyu Lu,et al.  Image-based indoor localization system based on 3D SfM model , 2014, Electronic Imaging.

[24]  Vineet R. Kamat,et al.  High-precision identification of contextual information in location-aware engineering applications , 2009, Adv. Eng. Informatics.

[25]  Markus König,et al.  Natural markers for augmented reality-based indoor navigation and facility maintenance , 2014 .

[26]  I-Tung Yang,et al.  An invisible head marker tracking system for indoor mobile augmented reality , 2013 .

[27]  Ming Jin,et al.  Automated mobile sensing: Towards high-granularity agile indoor environmental quality monitoring , 2018 .

[28]  Kwangmoon Cho,et al.  Development of Guide and Facility Management App by using Augmented Reality and QR Code , 2014 .

[29]  Torsten Sattler,et al.  Fast image-based localization using direct 2D-to-3D matching , 2011, 2011 International Conference on Computer Vision.

[30]  Fabio Bellavia,et al.  Robust Selective Stereo SLAM without Loop Closure and Bundle Adjustment , 2013, ICIAP.

[31]  Hyoungkwan Kim,et al.  Image retrieval using BIM and features from pretrained VGG network for indoor localization , 2018, Building and Environment.

[32]  Oh-Seong Kwon,et al.  A framework for proactive construction defect management using BIM, augmented reality and ontology-based data collection template , 2013 .

[33]  Mani Golparvar-Fard,et al.  Fast and scalable structure-from-motion based localization for high-precision mobile augmented reality systems , 2016, mUX: The Journal of Mobile User Experience.

[34]  J. M. M. Montiel,et al.  ORB-SLAM: A Versatile and Accurate Monocular SLAM System , 2015, IEEE Transactions on Robotics.

[35]  Yan Ke,et al.  PCA-SIFT: a more distinctive representation for local image descriptors , 2004, CVPR 2004.