On-Demand Urgent High Performance Computing Utilizing the Google Cloud Platform

In this paper we describe how high performance computing in the Google Cloud Platform can be utilized in an urgent and emergency situation to process large amounts of traffic data efficiently and on demand. Our approach provides a solution to an urgent need for disaster management using massive data processing and high performance computing. The traffic data used in this demonstration is collected from the public camera systems on Interstate highways in the Southeast United States. Our solution launches a parallel processing system that is the size of a Top 5 supercomputer using the Google Cloud Platform. Results show that the parallel processing system can be launched in a few hours, that it is effective at fast processing of high volume data, and can be de-provisioned in a few hours. We processed 211TB of video utilizing 6,227,593 core hours over the span of about eight hours with an average cost of around $0.008 per vCPU hour, which is less than the cost of many on-premise HPC systems.

[1]  Stanley T. Birchfield,et al.  A Taxonomy and Analysis of Camera Calibration Methods for Traffic Monitoring Applications , 2010, IEEE Transactions on Intelligent Transportation Systems.

[2]  Craig A. Jordan,et al.  Generic Incident Model for Investigating Traffic Incident Impacts on Evacuation Times in Large-Scale Emergencies , 2014 .

[3]  Dieter Kranzlmüller,et al.  Leveraging e-Infrastructures for Urgent Computing , 2013, ICCS.

[4]  Amy W. Apon,et al.  Sensitivity of Cluster File System Access to I/O Server Selection , 2002, 2nd IEEE/ACM International Symposium on Cluster Computing and the Grid (CCGRID'02).

[5]  Bartosz Balis,et al.  Execution Management and Efficient Resource Provisioning for Flood Decision Support , 2015, ICCS.

[6]  Dieter Kranzlmüller,et al.  Towards a General Definition of Urgent Computing , 2015, ICCS.

[7]  Hiroaki Kobayashi,et al.  Real-time tsunami inundation forecast system for tsunami disaster prevention and mitigation , 2018, The Journal of Supercomputing.

[8]  Baher Abdulhai,et al.  Harnessing the Power of HPC in Simulation and Optimization of Large Transportation Networks: Spatio-Temporal Traffic Management in the Greater Toronto Area , 2018, IEEE Intelligent Transportation Systems Magazine.

[9]  Stanley T. Birchfield,et al.  Real-Time Incremental Segmentation and Tracking of Vehicles at Low Camera Angles Using Stable Features , 2008, IEEE Transactions on Intelligent Transportation Systems.

[10]  Wayne A Sarasua,et al.  Automatic Camera Calibration Using Pattern Detection for Vision-Based Speed Sensing , 2008 .

[11]  Ivan Beschastnikh,et al.  SPRUCE: A System for Supporting Urgent High-Performance Computing , 2006, Grid-Based Problem Solving Environments.

[12]  Miron Livny,et al.  Pegasus, a workflow management system for science automation , 2015, Future Gener. Comput. Syst..

[13]  Alexander Herzog,et al.  Addressing the Challenges of Executing a Massive Computational Cluster in the Cloud , 2018, 2018 18th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing (CCGRID).

[14]  Alexander Boukhanovsky,et al.  Urgent Computing for Operational Storm Surge Forecasting in Saint-Petersburg , 2012, ICCS.