Visualizing Potential Transportation Demand From ETC Log Analysis Using ELK Stack

Traffic conditions are among the issues most concerned with the general public, and the freeway is a large-scale Internet-of-Things application. In addition to obtaining real-time road usage information, analysis of local road usage habits is crucial in evaluations of government policy implementation. Using road usage data provided by the electronic toll collection (ETC) system, we investigated the data on road usage history on the freeways. The ELK stack was employed to construct a platform for visualizing real-time road usage information and history in this article; the platform is named the local transportation knowledge (LTK) platform. By analyzing more than 500 million pieces of data, the LTK platform proposed in this article efficiently visualized road usage data and facilitated the acquirement of local road usage knowledge. We verified that residents of other counties and cities commuted to Taichung each day. We also discovered that a considerable number of Taichung city residents were employed in Hsinchu Science Park and commuted between the two places. The LTK platform can present real-time freeway traffic conditions, facilitate in-depth analysis of local road usage data, and provide data to verify the relevant information.

[1]  Lajiao Chen,et al.  Recent advance in earth observation big data for hydrology , 2018 .

[2]  Roy Fielding,et al.  Architectural Styles and the Design of Network-based Software Architectures"; Doctoral dissertation , 2000 .

[3]  Xiaoru Yuan,et al.  Visual Traffic Jam Analysis Based on Trajectory Data , 2013, IEEE Transactions on Visualization and Computer Graphics.

[4]  Christian Prehofer,et al.  Big data analytics architecture for real-time traffic control , 2017, 2017 5th IEEE International Conference on Models and Technologies for Intelligent Transportation Systems (MT-ITS).

[5]  Rajkumar Buyya,et al.  Big Data: Principles and Paradigms , 2016 .

[6]  Kai Lin,et al.  Vehicle Route Selection Based on Game Evolution in Social Internet of Vehicles , 2018, IEEE Internet of Things Journal.

[7]  Chao-Tung Yang,et al.  An Implementation of Scalable High Throughput Data Platform for Logging Semiconductor Testing Results , 2019, IEEE Access.

[8]  Klaus Moessner,et al.  An Ingestion and Analytics Architecture for IoT Applied to Smart City Use Cases , 2018, IEEE Internet of Things Journal.

[9]  Gerhard Nahler,et al.  Pearson Correlation Coefficient , 2020, Definitions.

[10]  Jun Bai,et al.  Feasibility analysis of big log data real time search based on Hbase and ElasticSearch , 2013, 2013 Ninth International Conference on Natural Computation (ICNC).

[11]  Jason J. Jung,et al.  Social big data: Recent achievements and new challenges , 2015, Information Fusion.

[12]  Bin Wu,et al.  Log analysis in cloud computing environment with Hadoop and Spark , 2013, 2013 5th IEEE International Conference on Broadband Network & Multimedia Technology.

[13]  Gang Xie,et al.  Air Quality Prediction: Big Data and Machine Learning Approaches , 2018 .

[14]  Feng Xia,et al.  LoTAD: long-term traffic anomaly detection based on crowdsourced bus trajectory data , 2017, World Wide Web.

[15]  Melody Moh,et al.  Detecting Web Attacks Using Multi-stage Log Analysis , 2016, 2016 IEEE 6th International Conference on Advanced Computing (IACC).

[16]  Manzhu Yu,et al.  A Cloud-Based Framework for Large-Scale Log Mining through Apache Spark and Elasticsearch , 2019, Applied Sciences.

[17]  Fei-Yue Wang,et al.  A Survey of Traffic Data Visualization , 2015, IEEE Transactions on Intelligent Transportation Systems.

[18]  Michael W. Godfrey,et al.  Mining modern repositories with elasticsearch , 2014, MSR 2014.

[19]  N. B. Anuar,et al.  The rise of "big data" on cloud computing: Review and open research issues , 2015, Inf. Syst..

[20]  Jules J. Berman,et al.  Principles of Big Data: Preparing, Sharing, and Analyzing Complex Information , 2013 .

[21]  Daniel E. O'Leary,et al.  Artificial Intelligence and Big Data , 2013, IEEE Intelligent Systems.

[22]  Shu-Kai S. Fan,et al.  Using machine learning and big data approaches to predict travel time based on historical and real-time data from Taiwan electronic toll collection , 2018, Soft Comput..

[23]  Liming Zhu,et al.  Detecting cloud provisioning errors using an annotated process model , 2013, MW4NextGen '13.

[24]  Adel M. Alimi,et al.  Big data analytics for logistics and transportation , 2015, 2015 4th International Conference on Advanced Logistics and Transport (ICALT).

[25]  Xiao Wang,et al.  Using Web data to enhance traffic situation awareness , 2014, 17th International IEEE Conference on Intelligent Transportation Systems (ITSC).

[26]  Anna V. Burunova.  The big data analysis , 2018, 2018 IEEE Conference of Russian Young Researchers in Electrical and Electronic Engineering (EIConRus).

[27]  Misha Kakkar,et al.  Geo-identification of web users through logs using ELK stack , 2016, 2016 6th International Conference - Cloud System and Big Data Engineering (Confluence).

[28]  Frédéric Beck,et al.  Powering Monitoring Analytics with ELK stack , 2015, AIMS 2015.

[29]  Muzameel Ahmed,et al.  Analysis of Logs by Using Logstash , 2016, FICTA.

[30]  Yu-Wei Chan,et al.  The Implementation of Wi-Fi Log Analysis System with ELK Stack , 2018 .

[31]  Widyawan,et al.  An evaluation of Twitter river and Logstash performances as elasticsearch inputs for social media analysis of Twitter , 2015, 2015 International Conference on Information & Communication Technology and Systems (ICTS).