FAS: A Flow Aware Scaling Mechanism for Stream Processing Platform Service based on LMS

Stream Processing Platform Service (SPPS is a service built on container cloud and implemented for purpose to develop a stream processing application with simple configuration. The service needs to provide scaling ability in order to adjust system capacity for dynamic incoming data volume. Data flow is a significant indicator for system load thus it becomes a fundamental factor for analyzing. Data flow prediction thus becomes important in order to improve Quality of Service (QoS) as well as optimize resource usage. In this paper, an approach applying Least Mean Squares (LMS) on data flow prediction with a scaling mechanism for system scaling is proposed. The algorithm takes period time and data flow into consideration to predicate the required resource for processing. After the data flow prediction is calculated, decision for new coming data is made, the service scales the processing cluster in advance for predicted volume. The experiment shows the method is effective for periodically changed data flow.

[1]  Shilpa Lakhina,et al.  Feature Reduction using Principal Component Analysis for Effective Anomaly – Based Intrusion Detection on NSL-KDD , 2010 .

[2]  Iftikhar Ahmad,et al.  Intrusion Detection Using PCA Based Modular Neural Network , 2012 .

[3]  Curtis E. Dyreson,et al.  Proceedings of the 2014 ACM SIGMOD International Conference on Management of Data , 2014, SIGMOD 2014.

[4]  Jian Tang,et al.  T-Storm: Traffic-Aware Online Scheduling in Storm , 2014, 2014 IEEE 34th International Conference on Distributed Computing Systems.

[5]  Jignesh M. Patel,et al.  Storm@twitter , 2014, SIGMOD Conference.

[6]  Neetesh Gupta,et al.  A Novel Approach to Intrusion Detection System using Rough Set Theory and Incremental SVM , 2011 .

[7]  Rung Ching Chen,et al.  Using Rough Set and Support Vector Machine for Network Intrusion Detection System , 2009, 2009 First Asian Conference on Intelligent Information and Database Systems.

[8]  Thomas S. Heinze,et al.  Online parameter optimization for elastic data stream processing , 2015, SoCC.

[9]  Yanhua Yu,et al.  Network traffic analysis and prediction based on APM , 2011, 2011 6th International Conference on Pervasive Computing and Applications.

[10]  Poo Kuan Hoong,et al.  Impact of utilizing forecasted network traffic for data transfers , 2011, 13th International Conference on Advanced Communication Technology (ICACT2011).

[11]  StonebrakerMichael,et al.  The 8 requirements of real-time stream processing , 2005 .

[12]  References , 1971 .

[13]  S. Douglas.  A family of normalized LMS algorithms , 1994, IEEE Signal Processing Letters.

[14]  Jay Kreps,et al.  Kafka : a Distributed Messaging System for Log Processing , 2011 .

[15]  Hao Chen,et al.  Prediction of traffic in a public safety network , 2006, 2006 IEEE International Symposium on Circuits and Systems.

[16]  Balasubramaniam Natarajan,et al.  GARCH — non-linear time series model for traffic modeling and prediction , 2008, NOMS 2008 - 2008 IEEE Network Operations and Management Symposium.