Cyber-physical system architecture for automating the mapping of truck loads to bridge behavior using computer vision in connected highway corridors

Abstract Bridges are critical components of highways ensuring traffic can efficiently travel over obstructions such as bodies of water, valleys, and other roads. Ensuring bridges are in sound structural condition is essential for safe and efficient highway operations. Structural health monitoring (SHM) systems designed to measure bridge responses have been developed to quantitatively track the health of bridges. More recently, SHM systems have also begun to integrate measurement of vehicular loads that create the responses measured. However, precise correlation of traffic loads to bridge responses remains a costly and technically difficult strategy. To address existing technical limitations, a cyber-physical system (CPS) framework is proposed to track truck loads in a highway corridor, to trigger SHM systems to record bridge responses, and to automate the linking of bridge response data with truck weights collected by weigh-in-motion (WIM) stations installed along the corridor but not collocated with the bridges. To link truck weights to bridge responses, computer vision methods based on convolutional neural networks (CNN) are used to automate the detection and reidentification of trucks using traffic cameras. The single-stage CNN object detector YOLO is trained using a customized dataset to identify trucks from camera images at each instrumentation site; high precision is obtained with the YOLO detector exceeding 95% average precision (AP) for an intersection over union (IOU) threshold of 0.75. To reidentify the same truck at different locations in the corridor, this study adopts a CNN-based encoder trained via a triplet network and a mutual nearest neighbor strategy using feature vectors extracted from images at each measurement location. The proposed reidentification method is implemented in the CPS cloud environment and obtains a F1-score of 0.97. The study also explores the triggering of bridge monitoring systems based on visual detection of trucks by a traffic camera installed upstream to the bridges. The triggering strategy proves to be highly efficient with 99% of the triggered data collection cycles capturing truck events at each bridge. To validate, the CPS architecture is implemented on a 20-mile highway corridor that has a WIM station already installed; four traffic cameras and two bridge SHM systems are installed along the corridor and integrated with a CPS architecture hosted on the cloud. In total, over 10,000 trucks are observed at all measurement locations over one year allowing peak bridge responses to be correlated to both measured truck weights and to one another.