Device Data Ingestion for Industrial Big Data Platforms with a Case Study

Despite having played a significant role in the Industry 4.0 era, the Internet of Things is currently faced with the challenge of how to ingest large-scale heterogeneous and multi-type device data. In response to this problem we present a heterogeneous device data ingestion model for an industrial big data platform. The model includes device templates and four strategies for data synchronization, data slicing, data splitting and data indexing, respectively. We can ingest device data from multiple sources with this heterogeneous device data ingestion model, which has been verified on our industrial big data platform. In addition, we present a case study on device data-based scenario analysis of industrial big data.

[1]  K. Shadan,et al.  Available online: , 2012 .

[2]  V. Chandrasekar,et al.  A peer-to-peer collaboration framework for multi-sensor data fusion , 2012, J. Netw. Comput. Appl..

[3]  Philip S. Yu,et al.  Early classification on time series , 2012, Knowledge and Information Systems.

[4]  Wei Fan,et al.  Mining big data: current status, and forecast to the future , 2013, SKDD.

[5]  Eamonn J. Keogh,et al.  Fast Shapelets: A Scalable Algorithm for Discovering Time Series Shapelets , 2013, SDM.

[6]  Himanshu Shah,et al.  Big Data Ingestion and Streaming Patterns , 2013 .

[7]  Harro Maas.  The Commercial and Political Atlas and Statistical Breviary , 2007 .

[8]  Rajiv Ranjan,et al.  Streaming Big Data Processing in Datacenter Clouds , 2014, IEEE Cloud Computing.

[9]  Shijun Liu,et al.  IBDP: An Industrial Big Data Ingestion and Analysis Platform and Case Studies , 2015, 2015 International Conference on Identification, Information, and Knowledge in the Internet of Things (IIKI).

[10]  Din J. Wasem,et al.  Mining of Massive Datasets , 2014 .

[11]  Christos Faloutsos,et al.  Efficient Similarity Search In Sequence Databases , 1993, FODO.

[12]  M. Waldrop,et al.  Community cleverness required , 2008, Nature.

[13]  Pascal Vasseur,et al.  Introduction to multi-sensor data fusion , 2004 .

[14]  Randy H. Katz,et al.  A view of cloud computing , 2010, CACM.

[15]  Eamonn J. Keogh.  Fast similarity search in the presence of longitudinal scaling in time series databases , 1997, Proceedings Ninth IEEE International Conference on Tools with Artificial Intelligence.

[16]  Xike Xie,et al.  Survey of real-time processing systems for big data , 2014, IDEAS.

[17]  Brian Hayes,et al.  What Is Cloud Computing? , 2019, Cloud Technologies.

[18]  Li Min-qiang,et al.  Time Series Segmentation Based on Series Importance Point , 2008 .

[19]  Ying Dai,et al.  Gobblin: Unifying Data Ingestion for Hadoop , 2015, Proc. VLDB Endow..

[20]  Eamonn J. Keogh,et al.  Logical-shapelets: an expressive primitive for time series classification , 2011, KDD.

[21]  Eamonn J. Keogh,et al.  Dimensionality Reduction for Fast Similarity Search in Large Time Series Databases , 2001, Knowledge and Information Systems.

[22]  Cyrus Shahabi,et al.  On the stationarity of multivariate time series for correlation-based data analysis , 2005, Fifth IEEE International Conference on Data Mining (ICDM'05).

[23]  Ada Wai-Chee Fu,et al.  Efficient time series matching by wavelets , 1999, Proceedings 15th International Conference on Data Engineering (Cat. No.99CB36337).

[24]  Alexandros Labrinidis,et al.  Challenges and Opportunities with Big Data , 2012, Proc. VLDB Endow..

[25]  Luiz André Barroso,et al.  The Datacenter as a Computer: An Introduction to the Design of Warehouse-Scale Machines , 2009, The Datacenter as a Computer: An Introduction to the Design of Warehouse-Scale Machines.

[26]  Uta Dresdner,et al.  Cloud Computing Methodology Systems And Applications , 2016 .

[27]  Zhihan Lv,et al.  A Self-Assessment Stereo Capture Model Applicable to the Internet of Things , 2015, Sensors.

[28]  John Gantz,et al.  The Digital Universe in 2020: Big Data, Bigger Digital Shadows, and Biggest Growth in the Far East , 2012 .

[29]  Hervé Jégou.  Efficient similarity search , 2018, Frontiers of Multimedia Research.

[30]  Lucy T. Nowell,et al.  ThemeRiver: visualizing theme changes over time , 2000, IEEE Symposium on Information Visualization 2000. INFOVIS 2000. Proceedings.

[31]  J. Mervis.  U.S. science policy. Agencies rally to tackle big data. , 2012, Science.

[32]  Syed Hassan Ahmed,et al.  A Novel Scheme for an Energy Efficient Internet of Things Based on Wireless Sensor Networks , 2015, Sensors.

[33]  Soumya Kanti Datta,et al.  Smart device sensing architectures and applications , 2013, 2013 International Computer Science and Engineering Conference (ICSEC).