An Accurate Ensemble Forecasting Approach for Highly Dynamic Cloud Workload With VMD and R-Transformer

To efficiently manage the cloud resources, improve the quality of service, and avoid the violations of Service-Level Agreement (SLA) agreements, it is very important to make accurate forecast for cloud workload. Prior works concerning cloud workload forecasting are mainly designed based on Recurrent Neural Networks (RNN). However, when it comes to a highly-dynamic cloud workload scenario where resource utilization changes faster and more frequently, these RNN-based methods are not effective in obtaining the linear and non-linear relationships and cannot give accurate forecast, because classic RNN has the problem of vanishing gradient. To address this issue, we propose an Ensemble Forecasting Approach (EFA) for highly-dynamic cloud workload by applying Variational Mode Decomposition (VMD) and R-Transformer. Specifically, to decrease the non-stationarity and high randomness of highly-dynamic cloud workload sequences, we decompose the workload into multiple Intrinsic Mode Functions (IMFs) by VMD. The IMFs are then imported into our ensemble forecasting module based on R-Transformer and Autoregressive model, in order to capture long-term dependencies and local non-linear relationship of workload sequences. The effectiveness and adaptability of proposed EFA is verified on real-world workload from Google and Alibaba cluster traces. Moreover, the performance evaluation results show that the EFA performs higher forecasting accuracy than prior related works over various forecasting time lengths for highly-dynamic cloud workload.

[1]  Sepp Hochreiter,et al.  The Vanishing Gradient Problem During Learning Recurrent Neural Nets and Problem Solutions , 1998, Int. J. Uncertain. Fuzziness Knowl. Based Syst..

[2]  Amir Hussain,et al.  A control theoretical view of cloud elasticity: taxonomy, survey and challenges , 2018, Cluster Computing.

[3]  Wei Guo,et al.  Short-Term Load Forecasting of Virtual Machines Based on Improved Neural Network , 2019, IEEE Access.

[4]  Francisco Herrera,et al.  A Forecasting Methodology for Workload Forecasting in Cloud Systems , 2018, IEEE Transactions on Cloud Computing.

[5]  Natalia Gimelshein,et al.  PyTorch: An Imperative Style, High-Performance Deep Learning Library , 2019, NeurIPS.

[6]  Bo Deng,et al.  Workload prediction for cloud computing elasticity mechanism , 2016, 2016 IEEE International Conference on Cloud Computing and Big Data Analysis (ICCCBDA).

[7]  Jian Sun,et al.  Deep Residual Learning for Image Recognition , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[8]  Zohra Amekraz,et al.  An Adaptive Workload Prediction Strategy for Non-Gaussian Cloud Service Using ARMA Model with Higher Order Statistics , 2018, 2018 IEEE 11th International Conference on Cloud Computing (CLOUD).

[9]  Yoshua Bengio,et al.  Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling , 2014, ArXiv.

[10]  Tarek El-Ghazawi,et al.  Towards Accurate Prediction for High-Dimensional and Highly-Variable Cloud Workloads with Deep Learning , 2020, IEEE Transactions on Parallel and Distributed Systems.

[11]  Jie Xu,et al.  Analysis, Modeling and Simulation of Workload Patterns in a Large-Scale Utility Cloud , 2014, IEEE Transactions on Cloud Computing.

[12]  Zhang Yuxian,et al.  Performance comparison of ANNs model with VMD for short‐term wind speed forecasting , 2018, IET Renewable Power Generation.

[13]  Wenbin Yao,et al.  Applying gated recurrent units pproaches for workload prediction , 2018, NOMS 2018 - 2018 IEEE/IFIP Network Operations and Management Symposium.

[14]  Ali Akbar Abdoos,et al.  A new intelligent method based on combination of VMD and ELM for short term wind power forecasting , 2016, Neurocomputing.

[15]  Dominique Zosso,et al.  Variational Mode Decomposition , 2014, IEEE Transactions on Signal Processing.

[16]  Razvan Pascanu,et al.  On the difficulty of training recurrent neural networks , 2012, ICML.

[17]  Ming-Wei Chang,et al.  BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding , 2019, NAACL.

[18]  Geoffrey E. Hinton,et al.  Layer Normalization , 2016, ArXiv.

[19]  Syed Riffat Ali.  Cloud Computing Reliability Analysis , 2019 .

[20]  Mostafa Ghobaei-Arani,et al.  A learning automata-based ensemble resource usage prediction algorithm for cloud computing environment , 2018, Future Gener. Comput. Syst..

[21]  Laurence T. Yang,et al.  An Efficient Deep Learning Model to Predict Cloud Workload for Industry Informatics , 2018, IEEE Transactions on Industrial Informatics.

[22]  Honghao Gao,et al.  A novel approach to workload prediction using attention-based LSTM encoder-decoder network in cloud environment , 2019, EURASIP J. Wirel. Commun. Netw..

[23]  Zhiwei Wang,et al.  R-Transformer: Recurrent Neural Network Enhanced Transformer , 2019, ArXiv.

[24]  Mansaf Alam,et al.  Analysis and Clustering of Workload in Google Cluster Trace Based on Resource Usage , 2015, 2016 IEEE Intl Conference on Computational Science and Engineering (CSE) and IEEE Intl Conference on Embedded and Ubiquitous Computing (EUC) and 15th Intl Symposium on Distributed Computing and Applications for Business Engineering (DCABES).

[25]  Guokun Lai,et al.  Modeling Long- and Short-Term Temporal Patterns with Deep Neural Networks , 2017, SIGIR.

[26]  Fábio Porto,et al.  Nonstationary time series transformation methods: An experimental review , 2019, Knowl. Based Syst..

[27]  Sam Jabbehdari,et al.  An autonomic resource provisioning approach for service-based cloud applications: A hybrid approach , 2018, Future Gener. Comput. Syst..

[28]  Lukasz Kaiser,et al.  Attention is All you Need , 2017, NIPS.

[29]  Jing Guo,et al.  Who Limits the Resource Efficiency of My Datacenter: An Analysis of Alibaba Datacenter Traces , 2019, 2019 IEEE/ACM 27th International Symposium on Quality of Service (IWQoS).

[30]  Jing Wu,et al.  A Type-Aware Workload Prediction Strategy for Non-stationary Cloud Service , 2017, 2017 IEEE 10th Conference on Service-Oriented Computing and Applications (SOCA).

[31]  Jürgen Schmidhuber,et al.  Long Short-Term Memory , 1997, Neural Computation.

[32]  Hoang Minh Nguyen,et al.  Host load prediction in cloud computing using Long Short-Term Memory Encoder–Decoder , 2019, The Journal of Supercomputing.

[33]  Maryam Amiri,et al.  Survey on prediction models of applications for resources provisioning in cloud , 2017, J. Netw. Comput. Appl..

[34]  Mohammad Masdari,et al.  A survey and classification of the workload forecasting methods in cloud computing , 2019, Cluster Computing.

[35]  Sunilkumar S. Manvi,et al.  Virtual resource prediction in cloud environment: A Bayesian approach , 2016, J. Netw. Comput. Appl..

[36]  Wenhu Chen,et al.  Enhancing the Locality and Breaking the Memory Bottleneck of Transformer on Time Series Forecasting , 2019, NeurIPS.

[37]  Rajkumar Buyya,et al.  Workload Prediction Using ARIMA Model and Its Impact on Cloud Applications’ QoS , 2015, IEEE Transactions on Cloud Computing.

[38]  Enda Barrett,et al.  Predicting host CPU utilization in cloud computing using recurrent neural networks , 2017, 2017 12th International Conference for Internet Technology and Secured Transactions (ICITST).

[39]  Yoshua Bengio,et al.  Learning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation , 2014, EMNLP.

[40]  Qihui Wu,et al.  A survey of machine learning for big data processing , 2016, EURASIP Journal on Advances in Signal Processing.

[41]  Parminder Singh,et al.  TASM: technocrat ARIMA and SVR model for workload prediction of web applications in cloud , 2018, Cluster Computing.

[42]  Prasad Saripalli,et al.  Load Prediction and Hot Spot Detection Models for Autonomic Cloud Computing , 2011, 2011 Fourth IEEE International Conference on Utility and Cloud Computing.