Context-based information generation for managing UAV-acquired data using image captioning

Abstract Unmanned aerial vehicles (UAVs) can efficiently collect image data representing various situations at construction sites, however, it requires a lot of time and cost to analyze it manually to retrieve useful information for on-site management. The paper proposes a methodology to generate time-spatial and visual context-based information from UAV-acquired data. This methodology generates textual information on the position, status, movement, color, and quantity of construction resources from site images using image captioning. Then, construction site images, text generated from them, and UAV flight data containing the latitude, longitude, date, and time of day, are systemized into a database. For evaluating the proposed methodology, data obtained by UAV at actual construction sites was used. Our methodology could predict textual information with a mean average precision of 43.52%, which is superior to those of existing methods.