Scene understanding in construction and buildings using image processing methods: A comprehensive review and a case study

Abstract Acquiring photos and videos has become a new norm in construction and building projects. However, imagery data is not utilized effectively due to the shortage of required skillsets in the industry and nonfamiliarity with classic image processing methods. Computer vision research in the context of construction and building has heavily focused on the interface between machine learning, and object tracking and activity recognition. Although positive results have been reported, namely improved productivity, safety and quality, implementations in the industry will not be immediate. Furthermore, algorithms such as convolutional neural networks (CNN), residual neural networks (ResNet) and recurrent neural networks (RNN) usually need to undergo extensive transfer learning in order to capture project-specific information in civil infrastructure engineering. This paper revisits classic image processing methods that can capture clues of site scenes with capability of high-level reasoning and inference. The work contributes to the body of knowledge by reviewing color, geometry and feature-based diagnostics in project environments.

[1]  Ioannis Brilakis,et al.  Machine Vision-Based Concrete Surface Quality Assessment , 2010 .

[2]  Fernanda Leite,et al.  Automated digital modeling of existing buildings: A review of visual object recognition methods , 2020 .

[3]  Pileun Kim,et al.  Framework for automated registration of UAV and UGV point clouds using local features in images , 2019, Automation in Construction.

[4]  Wen-Huang Cheng,et al.  A comparative study of data fusion for RGB-D based visual recognition , 2016, Pattern Recognit. Lett..

[5]  Shuihua Wang,et al.  RGB-D image-based detection of stairs, pedestrian crosswalks and traffic signs , 2014, J. Vis. Commun. Image Represent..

[6]  Mani Golparvar-Fard,et al.  Evaluation of image-based modeling and laser scanning accuracy for emerging automated performance monitoring techniques , 2011 .

[7]  Zhenhua Zhu,et al.  Image dataset development for measuring construction equipment recognition performance , 2014 .

[8]  Chollada Laofor,et al.  Defect detection and quantification system to support subjective visual quality inspection via a digital image processing: A tiling work case study , 2012 .

[9]  Jean-Laurent Hippolyte,et al.  Review: reconstruction of 3D building information models from 2D scanned plans , 2015 .

[10]  Suyoung Seo,et al.  Image Denoising and Refinement Based on an Iteratively Reweighted Least Squares Filter , 2020 .

[11]  Mehrdad Arashpour,et al.  The Less Agents, the More Schedule Reliability: Examination of Single-Point Responsibility Model in Design Management , 2019 .

[12]  Hyojoo Son,et al.  3D structural component recognition and modeling method using color and 3D data for construction progress monitoring , 2010 .

[13]  Hiroshi Hattori,et al.  Image-Based Monitoring of Open Gears of Movable Bridges for Condition Assessment and Maintenance Decision Making , 2015 .

[14]  Ali Karrech,et al.  Image Processing and Machine Learning to investigate fibre distribution on fibre-reinforced shotcrete Round Determinate Panels , 2018, Construction and Building Materials.

[15]  Young-Jin Cha,et al.  Vision-based detection of loosened bolts using the Hough transform and support vector machines , 2016 .

[16]  I-Ming Chen,et al.  Efficient Pose Estimation from Single RGB-D Image via Hough Forest with Auto-Context , 2018, 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS).

[17]  Gary A. Atkinson,et al.  Image segmentation of underfloor scenes using a mask regions convolutional neural network with two-stage transfer learning , 2020 .

[18]  D. Moher,et al.  Preferred reporting items for systematic reviews and meta-analyses: the PRISMA Statement , 2009, BMJ : British Medical Journal.

[19]  Eduardo Zalama,et al.  Automated Registration of 3D Scans Using Geometric Features and Normalized Color Data , 2013 .

[20]  Mohamed Marzouk,et al.  Quality analysis using three-dimensional modelling and image processing techniques , 2019 .

[21]  Yichang Tsai,et al.  Nonparametric Lens Debris Detection of Video Log Images Using Hysteresis Updating , 2012 .

[22]  Robert J. Thomas,et al.  Benchmarking Image Processing Algorithms for Unmanned Aerial System-Assisted Crack Detection in Concrete Structures , 2019, Infrastructures.

[23]  Kauser Ahmed,et al.  MULTI DOMAIN AUTOMATION SYSTEMS IN DOMESTIC ENVIRONMENTS , 2018 .

[24]  Mostafa Khanzadi,et al.  Lean design management using a gamified system , 2018 .

[25]  Yao Wang,et al.  Color-Guided Depth Recovery From RGB-D Data Using an Adaptive Autoregressive Model , 2014, IEEE Transactions on Image Processing.

[26]  Janaka Y. Ruwanpura,et al.  Automated Real-Time Monitoring System to Measure Shift Production of Tunnel Construction Projects , 2013 .

[27]  C. P. Kurian,et al.  Color characterization of multicolor multichip LED luminaire for indoor , 2018, Journal of Building Engineering.

[28]  Changyoon Kim,et al.  4D CAD model updating using image processing-based construction progress monitoring , 2013 .

[29]  Hyoungkwan Kim,et al.  Using Hue, Saturation, and Value Color Space for Hydraulic Excavator Idle Time Analysis , 2007 .

[30]  Peter E.D. Love,et al.  Computer vision applications in construction safety assurance , 2020 .

[31]  Robert J. Thomas,et al.  Comparison of deep convolutional neural networks and edge detectors for image-based crack detection in concrete , 2018, Construction and Building Materials.

[32]  Rui Ting,et al.  Vision technique for deflection measurements based on laser positioning , 2019, European Journal of Environmental and Civil Engineering.

[33]  Alex Albert,et al.  Automating and scaling personalized safety training using eye-tracking data , 2018, Automation in Construction.

[34]  Heng Li,et al.  Estimating Worker-Centric 3D Spatial Crowdedness for Construction Safety Management Using a Single 2D Camera , 2019, J. Comput. Civ. Eng..

[35]  Shuai Li,et al.  Vision-Based Excavator Detection and Tracking Using Hybrid Kinematic Shapes and Key Nodes , 2017, J. Comput. Civ. Eng..

[36]  James M. W. Brownjohn,et al.  Review of machine-vision based methodologies for displacement measurement in civil structures , 2018 .

[37]  Maria Q. Feng,et al.  Edge‐Enhanced Matching for Gradient‐Based Computer Vision Displacement Measurement , 2018, Comput. Aided Civ. Infrastructure Eng..

[38]  Mani Golparvar-Fard,et al.  Vision-based workface assessment using depth images for activity analysis of interior construction operations , 2014 .

[39]  Wei-Yen Hsu.  A Novel Image Registration Algorithm for Indoor and Built Environment Applications , 2015, Comput. Aided Civ. Infrastructure Eng..

[40]  Mahmoud R. Halfawy,et al.  Efficient Algorithm for Crack Detection in Sewer Images from Closed-Circuit Television Inspections , 2014 .

[41]  W. Park,et al.  Influence of bonded mortar of recycled concrete aggregates on interfacial characteristics – Porosity assessment based on pore segmentation from backscattered electron image analysis , 2019, Construction and Building Materials.

[42]  H. Son,et al.  Automated Color Model-Based Concrete Detection in Construction-Site Images by Using Machine Learning Algorithms , 2012, J. Comput. Civ. Eng..

[43]  Zhaohua Wang,et al.  Generalized Traffic Sign Detection Model for Developing a Sign Inventory , 2009 .

[44]  Qian Wang,et al.  Automatic classification of common building materials from 3D terrestrial laser scan data , 2020 .

[45]  Yang Yang,et al.  Deep-Learning-Based Bughole Detection for Concrete Surface Image , 2019, Advances in Civil Engineering.

[46]  Carl T. Haas,et al.  Pipe spool recognition in cluttered point clouds using a curvature-based shape descriptor , 2016 .

[47]  Gonzalo R. Arce,et al.  Multiresolution Information Mining for Pavement Crack Image Analysis , 2012 .

[48]  Garima Kadian,et al.  Review of Inpainting Techniques for UAV Images , 2019 .

[49]  Sehwan Kim,et al.  Vision-Based Natural Frequency Identification Using Laser Speckle Imaging and Parallel Computing , 2018, Comput. Aided Civ. Infrastructure Eng..

[50]  Tuan Ngo,et al.  Sensor-based safety management , 2020 .

[51]  Yichang James Tsai,et al.  Hybrid Active Contour-Incorporated Sign Detection Algorithm , 2012, J. Comput. Civ. Eng..

[52]  Luh-Maan Chang,et al.  Automated steel bridge coating rust defect recognition method based on color and texture feature , 2013 .

[53]  Aldo De La Haza,et al.  Assessment of concrete structures using the Mira and Eyecon ultrasonic shear wave devices and the SAFT-C image reconstruction technique , 2013 .

[54]  Xuefeng Zhao,et al.  Image-Based Concrete Crack Detection Using Convolutional Neural Network and Exhaustive Search Technique , 2019, Advances in Civil Engineering.

[55]  Hyojoo Son,et al.  A Comparative Study of Machine Learning Classification for Color-based Safety Vest Detection on Construction-Site Images , 2018, KSCE Journal of Civil Engineering.

[56]  Ioannis Brilakis,et al.  A vision-based method for on-road truck height measurement in proactive prevention of collision with overpasses and tunnels , 2015 .

[57]  Heng Li,et al.  Real-Time Alarming, Monitoring, and Locating for Non-Hard-Hat Use in Construction , 2019, Journal of Construction Engineering and Management.

[58]  Sangwook Lee,et al.  Automated recognition of surface defects using digital color image processing , 2006 .

[59]  Mani Golparvar-Fard,et al.  Automated 2D detection of construction equipment and workers from site video streams using histograms of oriented gradients and colors , 2013 .

[60]  Burcin Becerik-Gerber,et al.  Imaged-based verification of as-built documentation of operational buildings , 2012 .

[61]  Mehrdad Arashpour,et al.  A simulation model for investigation of operation of elevator’s up-peak , 2020, J. Simulation.

[63]  Deng-Fong Lin,et al.  Study the surface color of sewage sludge mortar at high temperature , 2007 .

[64]  Youngjib Ham,et al.  Vision-based nonintrusive context documentation for earthmoving productivity simulation , 2019, Automation in Construction.

[65]  Dino Bouchlaghem,et al.  Visualisation in architecture, engineering and construction (AEC) , 2005 .

[66]  Mohammad Ataei,et al.  Use of digital image processing techniques for evaluating wear of cemented carbide bits in rotary drilling , 2014 .

[67]  Jun Jo,et al.  A high precision crack classification system using multi-layered image processing and deep belief learning , 2020, Structure and Infrastructure Engineering.

[68]  Marziea Rahimi,et al.  3-D color histogram equalization by principal component analysis , 2016, J. Vis. Commun. Image Represent..

[69]  Fan Yang,et al.  A method to evaluate the segregation of compacted asphalt pavement by processing the images of paved asphalt mixture , 2019, Construction and Building Materials.

[70]  Q. P. Ha,et al.  Color tracking for multiple robot control using a system-on-programmable-chip , 2011 .

[71]  Mohamed Al-Hussein,et al.  A scientometric analysis and critical review of computer vision applications for construction , 2019, Automation in Construction.

[72]  Antonio Adán,et al.  Door detection in 3D coloured point clouds of indoor environments , 2018 .

[73]  Yashon O. Ouma,et al.  Pothole detection on asphalt pavements from 2D-colour pothole images using fuzzy c-means clustering and morphological reconstruction , 2017 .

[74]  Weili Fang,et al.  Ontology-Based Semantic Modeling of Knowledge in Construction: Classification and Identification of Hazards Implied in Images , 2020 .

[75]  Tarek Sayed,et al.  Framework for evaluating risk of limited sight distance for permitted left-turn movements: case study , 2016 .

[76]  Heng Li,et al.  Computer vision aided inspection on falling prevention measures for steeplejacks in an aerial environment , 2018, Automation in Construction.

[77]  Peter E.D. Love,et al.  Mapping computer vision research in construction: Developments, knowledge gaps and implications for research , 2019, Automation in Construction.

[78]  Mahmoud R. Halfawy,et al.  Integrated Vision-Based System for Automated Defect Detection in Sewer Closed Circuit Television Inspection Videos , 2015, J. Comput. Civ. Eng..

[79]  Nikos D. Lagaros,et al.  Low cost CPU–GPGPU parallel computing in real-world structural engineering , 2015 .

[80]  K. Sri Rama Krishna,et al.  Content based Image Retrieval based on Different Global and Local Color Histogram Methods: A Survey , 2017 .

[81]  Luh-Maan Chang,et al.  Human-visual-perception-like intensity recognition for color rust images based on artificial neural network , 2018, Automation in Construction.

[82]  Khaled Ksaibati,et al.  Image Retraining Using TensorFlow Implementation of the Pretrained Inception-v3 Model for Evaluating Gravel Road Dust , 2020, Journal of Infrastructure Systems.

[83]  Behrokh Khoshnevis,et al.  Computer vision for real-time extrusion quality monitoring and control in robotic construction , 2019, Automation in Construction.

[84]  Roman D. Hryciw,et al.  Identification and Characterization of Particle Shapes from Images of Sand Assemblies Using Pattern Recognition , 2018, J. Comput. Civ. Eng..

[85]  Hyung-Jo Jung,et al.  Quasi-autonomous bolt-loosening detection method using vision-based deep learning and image processing , 2019, Automation in Construction.

[86]  S. Sakthivel Murugan,et al.  Feature Extraction of Underwater Images by Combining Fuzzy C-Means Color Clustering and LBP Texture Analysis Algorithm with Empirical Mode Decomposition , 2019 .

[87]  Xiaochun Luo,et al.  Towards efficient and objective work sampling: Recognizing workers' activities in site surveillance videos with two-stream convolutional networks , 2018, Automation in Construction.

[88]  Yang Yang,et al.  Image analyses for video-based remote structure vibration monitoring system , 2016 .

[89]  Sungmoon Jung,et al.  Detection of hidden corrosion in metal roofing shingles utilizing infrared thermography , 2018, Journal of Building Engineering.

[90]  Junho Yeom,et al.  Cost-effective Framework for Rapid Underwater Mapping with Digital Camera and Color Correction Method , 2019, KSCE Journal of Civil Engineering.

[91]  Nhat-Duc Hoang,et al.  Image Processing-Based Classification of Asphalt Pavement Cracks Using Support Vector Machine Optimized by Artificial Bee Colony , 2018, J. Comput. Civ. Eng..

[92]  Ramón Moreno,et al.  Image segmentation on spherical coordinate representation of RGB colour space , 2012 .

[93]  Zhaozheng Hu,et al.  Generalized Image Recognition Algorithm for Sign Inventory , 2011 .