SMASH: A Cloud-Based Architecture for Big Data Processing and Visualization of Traffic Data

In recent times, big data has become a popular research topic and brought about a range of new challenges that must be tackled to support many commercial and research demands. The transport arena is one example that has much to benefit from big data capabilities in allowing to process voluminous amounts of data that is created in real time and in vast quantities. Tackling these big data issues requires capabilities not typically found in common Cloud platforms. This includes a distributed file system for capturing and storing data, a high performance computing engine able to process such large quantities of data, a reliable database system able to optimize the indexing and querying of the data, and geospatial capabilities to visualize the resultant analyzed data. In this paper we present SMASH, a generic and highly scalable Cloud-based architecture and its implementation that meets these many demands. We focus here specifically on the utilization of the SMASH software stack to process large scale traffic data for Adelaide and Victoria although we note that the solution can be applied to other big data processing areas. We provide performance results on SMASH and compare it with other big data solutions that have been developed.

[1]  Yannis Charalabidis,et al.  Benefits, Adoption Barriers and Myths of Open Data and Open Government , 2012, Inf. Syst. Manag..

[2]  Adam Jacobs,et al.  The pathologies of big data , 2009, Commun. ACM.

[3]  Jessica Anderson,et al.  Traffic signal timing determination: the Cabal model , 1997 .

[4]  L G Willumsen,et al.  SATURN - A SIMULATION-ASSIGNMENT MODEL FOR THE EVALUATION OF TRAFFIC MANAGEMENT SCHEMES , 1980 .

[5]  Divyakant Agrawal,et al.  Big data and cloud computing: current state and future opportunities , 2011, EDBT/ICDT '11.

[6]  Michael Schreckenberg,et al.  A verifiable simulation model for real-world microscopic traffic simulations , 2014, Simul. Model. Pract. Theory.

[7]  Anne E. Trefethen,et al.  The Data Deluge: An e-Science Perspective , 2003 .

[8]  Romain Billot,et al.  Microscopic cooperative traffic flow: calibration and simulation based on a next generation simulation dataset , 2014 .

[9]  C. Lynch.  Big data: How do your data grow? , 2008, Nature.

[10]  Sanjay Ghemawat,et al.  MapReduce: Simplified Data Processing on Large Clusters , 2004, OSDI.

[11]  L. Nelson.  Data, data everywhere. , 1997, Critical care medicine.

[12]  Howard Gobioff,et al.  The Google file system , 2003, SOSP '03.

[13]  Samuel Madden,et al.  From Databases to Big Data , 2012, IEEE Internet Comput..

[14]  Andrew O'Brien,et al.  SCATS Ramp Metering: Strategies, arterial integration and results , 2014, 17th International IEEE Conference on Intelligent Transportation Systems (ITSC).

[15]  P. Lowrie.  SCATS: Sydney Co-Ordinated Adaptive Traffic System: a traffic responsive method of controlling urban traffic , 1990 .

[16]  Wasan Pattara-Atikom,et al.  Social-based traffic information extraction and classification , 2011, 2011 11th International Conference on ITS Telecommunications.

[17]  Richard O. Sinnott,et al.  A Cloud-based Exploration of Open Data: Promoting Transparency and Accountability of the Federal Government of Australia , 2014, SIMBig.

[18]  Graham Currie,et al.  Prediction intervals to account for uncertainties in neural network predictions: Methodology and application in bus travel time prediction , 2011, Eng. Appl. Artif. Intell..

[19]  GhemawatSanjay,et al.  The Google file system , 2003 .

[20]  Richard O. Sinnott,et al.  Elastic Scaling of e-Infrastructures to Support Data-Intensive Research Collaborations , 2014, 2014 IEEE 10th International Conference on e-Science.

[21]  Hidetsugu Nanba,et al.  Extracting Transportation Information and Traffic Problems from Tweets during a Disaster , 2012 .

[22]  Yikai Gong,et al.  Identification of (near) Real-time Traffic Congestion in the Cities of Australia through Twitter , 2015, UCUI@CIKM.

[23]  Z QiuTony,et al.  Compatibility analysis of macroscopic and microscopic traffic simulation modeling , 2013 .

[24]  Karin Hedström,et al.  The story of the sixth myth of open data and open government , 2015 .