Data-driven scene parsing method for recognizing construction site objects in the whole image

Abstract Although efforts have been made for automated monitoring of construction sites, comprehensive understanding of a whole image remains to be a difficult task. Conventional vision-based monitoring methods have shortcomings in obtaining semantic information regarding an entire image because these methods are not scalable to the number of recognizable objects and training data. Most methods use a parametric model to recognize objects, involving cumbersome parameter tuning. This study presents the data-driven scene parsing method to recognize various objects in a construction site image. For identifying object information of a query image, the monitoring system retrieves the most relevant images to a query image using nearest neighbors and scale invariant feature transform flow matching and transfers relevant image labels to a query image. This study demonstrated the reasonable system performance in construction site images, recording 81.48% of average pixel-wise recognition rate with a small amount of similar images. The scene parsing method would enrich the raw information of a construction site image, thereby facilitating information use for various management applications.

[1]  Paul Fieguth,et al.  Automated detection of cracks in buried concrete pipe images , 2006 .

[2]  Silvio Savarese,et al.  Application of D4AR - A 4-Dimensional augmented reality model for automating construction progress monitoring data collection, processing and communication , 2009, J. Inf. Technol. Constr..

[3]  Ioannis Brilakis,et al.  Construction worker detection in video frames for initializing vision trackers , 2012 .

[4]  Lena Gorelick,et al.  Convexity Shape Prior for Segmentation , 2014, ECCV.

[5]  Carl T. Haas,et al.  State of research in automatic as-built modelling , 2015, Adv. Eng. Informatics.

[6]  Seokho Chi,et al.  Automated Object Identification Using Optical Video Cameras on Construction Sites , 2011, Comput. Aided Civ. Infrastructure Eng..

[7]  Thomas J. Armstrong,et al.  Motion Data-Driven Biomechanical Analysis during Construction Tasks on Sites , 2015 .

[8]  Patricio A. Vela,et al.  Construction performance monitoring via still images, time-lapse photos, and video streams: Now, tomorrow, and the future , 2015, Adv. Eng. Informatics.

[9]  Ioannis Brilakis,et al.  Detection of large-scale concrete columns for automated bridge inspection , 2010 .

[10]  Ioannis Brilakis,et al.  Parameter optimization for automated concrete detection in image data , 2010 .

[11]  Mani Golparvar-Fard,et al.  Visualization of construction progress monitoring with 4D simulation model overlaid on time-lapsed photographs , 2009 .

[12]  Ioannis Brilakis,et al.  Comparative study of vision tracking methods for tracking of construction site resources , 2011 .

[13]  Man-Woo Park,et al.  Hardhat-Wearing Detection for Enhancing On-Site Safety of Construction Workers , 2015 .

[14]  Patricio A. Vela,et al.  Personnel tracking on construction sites using video cameras , 2009, Adv. Eng. Informatics.

[15]  Manolis I. A. Lourakis,et al.  Automated as-built 3D reconstruction of civil infrastructure using computer vision: Achievements, opportunities, and challenges , 2015, Adv. Eng. Informatics.

[16]  Jie Gong,et al.  An object recognition, tracking, and contextual reasoning-based video interpretation method for rapid productivity analysis of construction operations , 2011 .

[17]  Sven J. Dickinson,et al.  Server-Customer Interaction Tracker: Computer Vision-Based System to Estimate Dirt-Loading Cycles , 2013 .

[18]  Hyoungkwan Kim,et al.  Data-driven scene parsing method for construction site monitoring , 2016 .

[19]  Andrey Dimitrov,et al.  Vision-based material recognition for automated monitoring of construction progress and generating building information modeling from unordered site image collections , 2014, Adv. Eng. Informatics.

[20]  Paul Fieguth,et al.  Segmentation of buried concrete pipe images , 2006 .

[21]  Ioannis Brilakis,et al.  Automated Brick Counting for Façade Construction Progress Estimation , 2015, J. Comput. Civ. Eng..

[22]  Mani Golparvar-Fard,et al.  Automated Progress Monitoring Using Unordered Daily Construction Photographs and IFC-Based Building Information Models , 2015, J. Comput. Civ. Eng..

[23]  Hyoungkwan Kim,et al.  Using Hue, Saturation, and Value Color Space for Hydraulic Excavator Idle Time Analysis , 2007 .

[24]  Yozo Fujino,et al.  Concrete Crack Detection by Multiple Sequential Image Filtering , 2012, Comput. Aided Civ. Infrastructure Eng..

[25]  Brenda McCabe,et al.  Automated Visual Recognition of Dump Trucks in Construction Videos , 2012, J. Comput. Civ. Eng..

[26]  Brenda McCabe,et al.  Part based model and spatial–temporal reasoning to recognize hydraulic excavators in construction images and videos , 2012 .

[27]  Christian Koch,et al.  Pothole detection in asphalt pavement images , 2011, Adv. Eng. Informatics.

[28]  Hyojoo Son,et al.  3D structural component recognition and modeling method using color and 3D data for construction progress monitoring , 2010 .

[29]  Lucio Soibelman,et al.  Construction site image retrieval based on material cluster recognition , 2006, Adv. Eng. Informatics.

[30]  Juan Carlos Niebles,et al.  Vision-based action recognition of earthmoving equipment using spatio-temporal features and support vector machine classifiers , 2013, Adv. Eng. Informatics.

[31]  Frédéric Bosché,et al.  As-built data acquisition and its use for production monitoring and automated layout in civil infrastructure: a survey , 2015 .

[32]  Zhongke Shi,et al.  Tracking multiple workers on construction sites using video cameras , 2010, Adv. Eng. Informatics.

[33]  Christiane Fellbaum,et al.  Book Reviews: WordNet: An Electronic Lexical Database , 1999, CL.

[34]  Kinam Kim,et al.  Vision-Based Object-Centric Safety Assessment Using Fuzzy Inference: Monitoring Struck-By Accidents with Moving Objects , 2016, J. Comput. Civ. Eng..

[35]  Seokho Chi,et al.  Image-Based Safety Assessment: Automated Spatial Safety Risk Identification of Earthmoving and Surface Mining Activities , 2012 .

[36]  Lena Gorelick,et al.  GrabCut in One Cut , 2013, 2013 IEEE International Conference on Computer Vision.

[37]  Zhongke Shi,et al.  A performance evaluation of vision and radio frequency tracking methods for interacting workforce , 2011, Adv. Eng. Informatics.

[38]  Jie Gong,et al.  Computer Vision-Based Video Interpretation Model for Automated Productivity Analysis of Construction Operations , 2010 .

[39]  Carlos H. Caldas,et al.  Real-Time Three-Dimensional Occupancy Grid Modeling for the Detection and Tracking of Construction Resources , 2007 .

[40]  H. Son,et al.  Automated Color Model-Based Concrete Detection in Construction-Site Images by Using Machine Learning Algorithms , 2012, J. Comput. Civ. Eng..

[41]  Jochen Teizer,et al.  Status quo and open challenges in vision-based sensing and tracking of temporary resources on infrastructure construction sites , 2015, Adv. Eng. Informatics.

[42]  David S. Citrin,et al.  An analogy between various machine-learning techniques for detecting construction materials in digital images , 2015, KSCE Journal of Civil Engineering.

[43]  Seokho Chi,et al.  A Methodology for Object Identification and Tracking in Construction Based on Spatial Modeling and Image Matching Techniques , 2009, Comput. Aided Civ. Infrastructure Eng..

[44]  Cordelia Schmid,et al.  Transformation Pursuit for Image Classification , 2014, 2014 IEEE Conference on Computer Vision and Pattern Recognition.

[45]  Svetlana Lazebnik,et al.  Superparsing - Scalable Nonparametric Image Parsing with Superpixels , 2010, International Journal of Computer Vision.

[46]  Antonio Torralba,et al.  LabelMe: A Database and Web-Based Tool for Image Annotation , 2008, International Journal of Computer Vision.

[47]  Ioannis Brilakis,et al.  Visual retrieval of concrete crack properties for automated post-earthquake structural safety evaluation , 2011 .

[48]  Christian Koch,et al.  Automated Pothole Distress Assessment Using Asphalt Pavement Video Data , 2013 .

[49]  David G. Lowe,et al.  Distinctive Image Features from Scale-Invariant Keypoints , 2004, International Journal of Computer Vision.

[50]  Paul Fieguth,et al.  Neuro-fuzzy network for the classification of buried pipe defects , 2006 .

[51]  Feniosky Peña-Mora,et al.  Vision-Based Detection of Unsafe Actions of a Construction Worker: Case Study of Ladder Climbing , 2013, J. Comput. Civ. Eng..

[52]  Mani Golparvar-Fard,et al.  Automated 2D detection of construction equipment and workers from site video streams using histograms of oriented gradients and colors , 2013 .

[53]  Ioannis K. Brilakis,et al.  Automated vision tracking of project related entities , 2011, Adv. Eng. Informatics.

[54]  Changmin Kim,et al.  Classification of major construction materials in construction environments using ensemble classifiers , 2014, Adv. Eng. Informatics.

[55]  Seung Heon Han,et al.  Object recognition in construction-site images using 3D CAD-based filtering , 2010 .

[56]  SangUk Han,et al.  A vision-based motion capture and recognition framework for behavior-based safety management , 2013 .

[57]  Richard Szeliski,et al.  A Comparative Study of Energy Minimization Methods for Markov Random Fields with Smoothness-Based Priors , 2008, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[58]  Jochen Teizer,et al.  Mobile 3D mapping for surveying earthwork projects using an Unmanned Aerial Vehicle (UAV) system , 2014 .

[59]  Christian Koch,et al.  Three-Dimensional Tracking of Construction Resources Using an On-Site Camera System , 2012, J. Comput. Civ. Eng..

[60]  AhmedMahmoud,et al.  Using digital photogrammetry for pipe-works progress tracking11This paper is one of a selection of papers in this Special Issue on Construction Engineering and Management. , 2012 .

[61]  Amin Hammad,et al.  Automated annotation for visual recognition of construction resources using synthetic images , 2016 .

[62]  Mani Golparvar-Fard,et al.  Segmentation and recognition of roadway assets from car-mounted camera video streams using a scalable non-parametric image parsing method , 2015 .

[63]  Paul W. Fieguth,et al.  A review on computer vision based defect detection and condition assessment of concrete and asphalt civil infrastructure , 2015, Adv. Eng. Informatics.

[64]  Antonio Torralba,et al.  Nonparametric Scene Parsing via Label Transfer , 2011, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[65]  Janaka Y. Ruwanpura,et al.  Automated Real-Time Monitoring System to Measure Shift Production of Tunnel Construction Projects , 2013 .

[66]  Feniosky Peña-Mora,et al.  Comparative Study of Motion Features for Similarity-Based Modeling and Classification of Unsafe Actions in Construction , 2014, J. Comput. Civ. Eng..

[67]  Changmin Kim,et al.  Rapid and automated determination of rusted surface areas of a steel bridge for robotic maintenance systems , 2014 .

[68]  Lucio Soibelman,et al.  Material-Based Construction Site Image Retrieval , 2005 .

[69]  Mani Golparvar-Fard,et al.  Appearance-based material classification for monitoring of operation-level construction progress using 4D BIM and site photologs , 2015 .

[70]  Vladimir Kolmogorov,et al.  "GrabCut": interactive foreground extraction using iterated graph cuts , 2004, ACM Trans. Graph..

[71]  Antonio Criminisi,et al.  TextonBoost for Image Understanding: Multi-Class Object Recognition and Segmentation by Jointly Modeling Texture, Layout, and Context , 2007, International Journal of Computer Vision.

[72]  Antonio Torralba,et al.  Modeling the Shape of the Scene: A Holistic Representation of the Spatial Envelope , 2001, International Journal of Computer Vision.

[73]  Antonio Torralba,et al.  SIFT Flow: Dense Correspondence across Scenes and Its Applications , 2011, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[74]  Changyoon Kim,et al.  4D CAD model updating using image processing-based construction progress monitoring , 2013 .

[75]  SangHyun Lee,et al.  Computer vision techniques for construction safety and health monitoring , 2015, Adv. Eng. Informatics.

[76]  Kumar,et al.  Neural Networks a Classroom Approach , 2004 .

[77]  Marie-Pierre Jolly,et al.  Interactive graph cuts for optimal boundary & region segmentation of objects in N-D images , 2001, Proceedings Eighth IEEE International Conference on Computer Vision. ICCV 2001.