Ahab: A cloud‐based distributed big data analytics framework for the Internet of Things

Smart city applications generate large amounts of operational data during their execution, such as information from infrastructure monitoring, performance and health events from used toolsets, and application execution logs. These data streams contain vital information about the execution environment that can be used to fine‐tune or optimize different layers of a smart city application infrastructure. Current approaches do not sufficiently address the efficient collection, processing, and storage of this information in the smart city domain. In this paper, we present Ahab, a generic, scalable, and fault‐tolerant data processing framework based on the cloud that allows operators to perform online and offline analyses on gathered data to better understand and optimize the behavior of the available smart city infrastructure. Ahab is designed for easy integration of new data sources, provides an extensible API to perform custom analysis tasks, and a domain‐specific language to define adaptation rules based on analysis results. We demonstrate the feasibility of the proposed approach using an example application for autonomous intersection management in smart city environments. Our framework is able to autonomously optimize application deployment topologies by distributing processing load over available infrastructure resources when necessary based on both online analysis of the current state of the environment and patterns learned from historical data. Copyright © 2016 John Wiley & Sons, Ltd.

[1]  Schahram Dustdar,et al.  LEONORE -- Large-Scale Provisioning of Resource-Constrained IoT Deployments , 2015, 2015 IEEE Symposium on Service-Oriented System Engineering.

[2]  Divyakant Agrawal,et al.  Big data and cloud computing: current state and future opportunities , 2011, EDBT/ICDT '11.

[3]  Fatos Xhafa,et al.  Processing and Analytics of Big Data Streams with Yahoo!S4 , 2015, 2015 IEEE 29th International Conference on Advanced Information Networking and Applications.

[4]  Eric Gossett,et al.  Big Data: A Revolution That Will Transform How We Live, Work, and Think , 2015 .

[5]  Wilson C. Hsieh,et al.  Bigtable: A Distributed Storage System for Structured Data , 2006, TOCS.

[6]  Albert Y. Zomaya,et al.  Recent advances in autonomic provisioning of big data applications on clouds , 2015, IEEE Trans. Cloud Comput..

[7]  Scott Shenker,et al.  Discretized streams: fault-tolerant streaming computation at scale , 2013, SOSP.

[8]  Yadira Espinal.  Viktor Mayer-Schonberger and Kenneth Cukier, Big Data: A Revolution That Will Transform How We Live, Work and Think , 2013 .

[9]  Ying Xing,et al.  The Design of the Borealis Stream Processing Engine , 2005, CIDR.

[10]  Thomas S. Heinze,et al.  Latency-aware elastic scaling for distributed data stream processing systems , 2014, DEBS '14.

[11]  Schahram Dustdar,et al.  Generic event‐based monitoring and adaptation methodology for heterogeneous distributed systems , 2014, Softw. Pract. Exp..

[12]  Michael L. Brodie,et al.  The meaningful use of big data: four perspectives -- four challenges , 2012, SGMD.

[13]  Peter Stone,et al.  Autonomous Intersection Management: Multi-intersection optimization , 2011, 2011 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[14]  Rajkumar Buyya,et al.  Big Data computing and clouds: Trends and future directions , 2013, J. Parallel Distributed Comput..

[15]  Qiming Chen,et al.  Cut-and-Rewind: Extending Query Engine for Continuous Stream Analytics , 2015, Trans. Large Scale Data Knowl. Centered Syst..

[16]  Schahram Dustdar,et al.  Distributed continuous queries over Web service event streams , 2011, 2011 7th International Conference on Next Generation Web Services Practices.

[17]  Schahram Dustdar,et al.  DIANE - Dynamic IoT Application Deployment , 2015, 2015 IEEE International Conference on Mobile Services.

[18]  Antonio Iera,et al.  The Internet of Things: A survey , 2010, Comput. Networks.

[19]  Prashant Malik,et al.  Cassandra: a decentralized structured storage system , 2010, OPSR.

[20]  Sam Newman,et al.  Building Microservices , 2015 .

[21]  Antonio Puliafito,et al.  Enabling the Cloud of Things , 2012, 2012 Sixth International Conference on Innovative Mobile and Internet Services in Ubiquitous Computing.

[22]  Schahram Dustdar,et al.  Esc: Towards an Elastic Stream Computing Platform for the Cloud , 2011, 2011 IEEE 4th International Conference on Cloud Computing.

[23]  Jinjun Chen,et al.  A Big Picture of Integrity Verification of Big Data in Cloud Computing , 2015, Handbook on Data Centers.

[24]  BonczPeter,et al.  The meaningful use of big data , 2012 .

[25]  Keqiu Li,et al.  Big Data Processing in Cloud Computing Environments , 2012, 2012 12th International Symposium on Pervasive Systems, Algorithms and Networks.

[26]  Antonio Puliafito,et al.  AllJoyn Lambda: An architecture for the management of smart environments in IoT , 2014, 2014 International Conference on Smart Computing Workshops.

[27]  Albert Y. Zomaya,et al.  Big Data Privacy in the Internet of Things Era , 2014, IT Professional.

[28]  Yanlei Diao,et al.  High-performance complex event processing over streams , 2006, SIGMOD Conference.

[29]  Srinivas Aluru,et al.  Editorial: Scalable Systems for Big Data Management and Analytics , 2015, J. Parallel Distributed Comput..

[30]  Daniel M. Batista,et al.  A Survey of Large Scale Data Management Approaches in Cloud Environments , 2011, IEEE Communications Surveys & Tutorials.

[31]  Divyakant Agrawal,et al.  Big data and cloud computing , 2010, Proc. VLDB Endow..

[32]  Christof Fetzer,et al.  Auto-scaling techniques for elastic data stream processing , 2014, 2014 IEEE 30th International Conference on Data Engineering Workshops.

[33]  Lida Xu,et al.  The internet of things: a survey , 2014, Information Systems Frontiers.

[34]  Surajit Chaudhuri,et al.  What next?: a half-dozen data management research goals for big data and the cloud , 2012, PODS '12.

[35]  Werner Vogels,et al.  Dynamo: amazon's highly available key-value store , 2007, SOSP.