Estimation of crowd flow and load on pedestrian bridges using machine learning with sensor fusion

Abstract Unfortunate tragedies have been the result of crowd behavior in high-density pedestrian flows as a reaction to an incident aggravating the conditions and resulting in trampling and crushing situations. Therefore, it is important to monitor such crowd motion for danger warning and prevention. In this work, we utilize a recently introduced framework for simultaneous crowd and structural monitoring based on a novel combination of sensing technologies that includes the employment of structurally mounted Fiber Bragg Grating (FBG) Fiber Optic Sensors (FOSs) in conjunction with individually held wearable sensing devices incorporating Inertial Measurement Units (IMUs). We propose effective approaches for the estimation of crowd flow and load on pedestrian bridges using advanced machine learning techniques with sensor data extracted from both structural sensors and wearable devices. A key novelty of the approaches presented is the utilization of sensor fusion at both the input and feature level within the machine learning methods used. The machine learning approaches include Convolutional Neural Networks (CNN) and Support Vector Machines (SVM); they are first applied on each of the sensing sources, FOSs and IMUs, individually and then extended for multi-modal data fusion at both the input and feature levels. Data fusion enriches the learning models and enhances the system efficiency and robustness against noise and input data time-shifting errors. The proposed approaches are validated using experimental measurements from a crowd replication scenario run on a small-scaled test bridge. The trained machine learning models demonstrate effectiveness in crowd attribute classification for motion speed and load designation, along with a regression model for load estimation. Results show that the proposed monitoring solution is highly effective with peak testing accuracy for single class motion speed classification at 98%, multi-class motion speed and load characterization classification at 91%, and percentage error for load estimation regression reaching a minimum of 9%.

[1]  Vishal M. Patel,et al.  CNN-Based cascaded multi-task learning of high-level prior and density estimation for crowd counting , 2017, 2017 14th IEEE International Conference on Advanced Video and Signal Based Surveillance (AVSS).

[2]  Maria Q. Feng,et al.  Biomechanically influenced mobile and participatory pedestrian data for bridge monitoring , 2017, Int. J. Distributed Sens. Networks.

[3]  Shenghui Zhao,et al.  A Comparative Study on Human Activity Recognition Using Inertial Sensors in a Smartphone , 2016, IEEE Sensors Journal.

[4]  Amir Hossein Alavi,et al.  An overview of smartphone technology for citizen-centered, real-time and scalable civil infrastructure monitoring , 2019, Future Gener. Comput. Syst..

[5]  Sung-Bae Cho,et al.  Human activity recognition with smartphone sensors using deep learning neural networks , 2016, Expert Syst. Appl..

[6]  Mao Ye,et al.  Fast crowd density estimation with convolutional neural networks , 2015, Eng. Appl. Artif. Intell..

[7]  Hao Wang,et al.  Use of FBG Sensors for SHM in Aerospace Structures , 2012 .

[8]  Mingchu Li,et al.  Initial Validation of Mobile-Structural Health Monitoring Method Using Smartphones , 2015, Int. J. Distributed Sens. Networks.

[9]  Senem Velipasalar,et al.  A Survey on Activity Detection and Classification Using Wearable Sensors , 2017, IEEE Sensors Journal.

[10]  Monji Kherallah,et al.  A New Design Based-SVM of the CNN Classifier Architecture with Dropout for Offline Arabic Handwritten Recognition , 2016, ICCS.

[11]  Ken Eason,et al.  Factors influencing experience in crowds - The participant perspective. , 2017, Applied ergonomics.

[12]  Y. LeCun,et al.  Learning methods for generic object recognition with invariance to pose and lighting , 2004, Proceedings of the 2004 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2004. CVPR 2004..

[13]  Maria Q. Feng,et al.  Direction-sensitive smart monitoring of structures using heterogeneous smartphone sensor data and coordinate system transformation , 2017 .

[14]  Nasser Kehtarnavaz,et al.  A survey of depth and inertial sensor fusion for human action recognition , 2015, Multimedia Tools and Applications.

[15]  Mohamed Abdel-Mottaleb,et al.  Discriminant Correlation Analysis: Real-Time Feature Level Fusion for Multimodal Biometric Recognition , 2016, IEEE Transactions on Information Forensics and Security.

[16]  Sergio A. Velastin,et al.  Crowd analysis: a survey , 2008, Machine Vision and Applications.

[17]  Jerome M. Chertkoff,et al.  Don't Panic: The Psychology of Emergency Egress and Ingress , 1999 .

[18]  Maria Q. Feng,et al.  Citizen Sensors for SHM: Towards a Crowdsourcing Platform , 2015, Sensors.

[19]  Özlem Durmaz Incel,et al.  User, device and orientation independent human activity recognition on mobile phones: challenges and a proposal , 2013, UbiComp.

[20]  Vassilis Kostopoulos,et al.  Health Monitoring of Aerospace Structures Using Fibre Bragg Gratings Combined with Advanced Signal Processing and Pattern Recognition Techniques , 2012 .

[21]  Ashish Verma,et al.  A review of studies on understanding crowd dynamics in the context of crowd safety in mass religious gatherings , 2017 .

[22]  Peter Crossley,et al.  Time Synchronization for Transmission Substations Using GPS and IEEE 1588 , 2016 .

[23]  Ayan Sadhu,et al.  A literature review of next‐generation smart sensing technology in structural health monitoring , 2019, Structural Control and Health Monitoring.

[24]  Hao Wu,et al.  Multi-sensor Golf Swing Classification Using Deep CNN , 2017, IIKI.

[25]  Dirk Helbing,et al.  Crowd disasters as systemic failures: analysis of the Love Parade disaster , 2012, EPJ Data Science.

[26]  Vishal M. Patel,et al.  A Survey of Recent Advances in CNN-based Single Image Crowd Counting and Density Estimation , 2017, Pattern Recognit. Lett..

[27]  Shenghua Gao,et al.  Single-Image Crowd Counting via Multi-Column Convolutional Neural Network , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[28]  Rachel Cardell-Oliver,et al.  Occupancy Estimation Using a Low-Pixel Count Thermal Imager , 2016, IEEE Sensors Journal.

[29]  Gary M. Weiss,et al.  Activity recognition using cell phone accelerometers , 2011, SKDD.

[30]  Lei Gao,et al.  Activity recognition using dynamic multiple sensor fusion in body sensor networks , 2012, 2012 Annual International Conference of the IEEE Engineering in Medicine and Biology Society.

[31]  Simon Haykin,et al.  GradientBased Learning Applied to Document Recognition , 2001 .

[32]  Mingchu Li,et al.  Experimental Research on Quick Structural Health Monitoring Technique for Bridges Using Smartphone , 2016 .

[33]  Theodoros H. Loutas,et al.  Intelligent health monitoring of aerospace composite structures based on dynamic strain measurements , 2012, Expert Syst. Appl..

[34]  Yoshua Bengio,et al.  Gradient-based learning applied to document recognition , 1998, Proc. IEEE.

[35]  Luca Maria Gambardella,et al.  Flexible, High Performance Convolutional Neural Networks for Image Classification , 2011, IJCAI.

[36]  Marc'Aurelio Ranzato,et al.  Unsupervised Learning of Invariant Feature Hierarchies with Applications to Object Recognition , 2007, 2007 IEEE Conference on Computer Vision and Pattern Recognition.

[37]  Paul Lukowicz,et al.  Activity Recognition of Assembly Tasks Using Body-Worn Microphones and Accelerometers , 2006, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[38]  Chau Yuen,et al.  Sensor Fusion for Public Space Utilization Monitoring in a Smart City , 2017, IEEE Internet of Things Journal.

[39]  Amol P. Bhondekar,et al.  A review on technological advancements in crowd management , 2018, J. Ambient Intell. Humaniz. Comput..

[40]  Alexis Boukouvalas,et al.  Predicting room occupancy with a single passive infrared (PIR) sensor through behavior extraction , 2016, UbiComp.

[41]  Samir Mustapha,et al.  Multi-Source Data Fusion for Classification of Surface Cracks in Steel Pipes , 2018 .

[42]  Shamim N. Pakzad,et al.  Smartphone data streams for bridge health monitoring , 2017 .

[43]  Samir Mustapha,et al.  Joint Crowd Management and Structural Health Monitoring Using Fiber Optic and Wearable Sensing , 2019, IEEE Communications Magazine.

[44]  Adil Mehmood Khan,et al.  Activity Recognition on Smartphones via Sensor-Fusion and KDA-Based SVMs , 2014, Int. J. Distributed Sens. Networks.

[45]  Moncef Gabbouj,et al.  Real-time vibration-based structural damage detection using one-dimensional convolutional neural networks , 2017 .

[46]  Paul Lukowicz,et al.  Using Mobile Technology and a Participatory Sensing Approach for Crowd Monitoring and Management During Large-Scale Mass Gatherings , 2013 .

[47]  Amin Al-Habaibeh,et al.  Real-time crowd density mapping using a novel sensory fusion model of infrared and visual systems , 2013 .

[48]  Naser Jawas,et al.  CNN and SVM Based Classifier Comparation to Detect Lung Nodule In Computed Tomography Images , 2018 .