IBDP: An Industrial Big Data Ingestion and Analysis Platform and Case Studies

The Internet of Things (IoT) brings traditional Internet industry and society with new trends and promising technologies. For industrial information with high amount and renewal speed characteristics, resulting in difficult data ingestion and analysis, this paper presented an Industrial Big Data ingestion and analysis Platform (IBDP). In the platform, we integrated HDFS, Spark, Hive, HBase, Flume, Sqoop, OpenStack etc. It works well for industrial data ingestion and analysis. In addition, we report some case studies on industrial big data processing flows respect to different data types.

[1]  Luiz André Barroso,et al.  The Datacenter as a Computer: An Introduction to the Design of Warehouse-Scale Machines , 2009, The Datacenter as a Computer: An Introduction to the Design of Warehouse-Scale Machines.

[2]  Uta Dresdner,et al.  Cloud Computing Methodology Systems And Applications , 2016 .

[3]  Wilson C. Hsieh,et al.  Bigtable: A Distributed Storage System for Structured Data , 2006, TOCS.

[4]  Rajiv Ranjan,et al.  Cloud Computing: Methodology, Systems, and Applications , 2011 .

[5]  Randy H. Katz,et al.  A view of cloud computing , 2010, CACM.

[6]  Scott Shenker,et al.  Fast and Interactive Analytics over Hadoop Data with Spark , 2012, login Usenix Mag..

[7]  Sanjay Ghemawat,et al.  MapReduce: Simplified Data Processing on Large Clusters , 2004, OSDI.

[8]  Rajiv Ranjan,et al.  Streaming Big Data Processing in Datacenter Clouds , 2014, IEEE Cloud Computing.

[9]  Xike Xie,et al.  Survey of real-time processing systems for big data , 2014, IDEAS.

[10]  Mohamed Jmaiel,et al.  A Comparative Study of the Current Cloud Computing Technologies and Offers , 2011, 2011 First International Symposium on Network Cloud Computing and Applications.

[11]  Mario Zagar,et al.  Evaluating open-source cloud computing solutions , 2011, 2011 Proceedings of the 34th International Convention MIPRO.

[12]  J. Mervis.  U.S. science policy. Agencies rally to tackle big data. , 2012, Science.

[13]  Din J. Wasem,et al.  Mining of Massive Datasets , 2014 .

[14]  Xindong Wu,et al.  Data mining with big data , 2014, IEEE Transactions on Knowledge and Data Engineering.

[15]  John Gantz,et al.  The Digital Universe in 2020: Big Data, Bigger Digital Shadows, and Biggest Growth in the Far East , 2012 .

[16]  Wei Fan,et al.  Mining big data: current status, and forecast to the future , 2013, SKDD.

[17]  Mohsine Eleuldj,et al.  OpenStack: Toward an Open-source Solution for Cloud Computing , 2012 .

[18]  Alexandros Labrinidis,et al.  Challenges and Opportunities with Big Data , 2012, Proc. VLDB Endow..