Anomaly-Tolerant Traffic Matrix Estimation via Prior Information Guided Matrix Completion

Network management, planning, and optimization rely on accurate and complete traffic measurement. However, anomalies and missing data are inevitable in direct traffic measurement because of high measurement costs and unreliable network transport protocols. Existing traffic matrix estimation approaches concern only the outlier and ignore the structural anomaly, which often degrades the estimation accuracy drastically. To address this challenge, an anomaly-tolerant traffic matrix estimation approach called Simultaneously Estimate Traffic Matrix and Detect Anomaly (SETMADA) is presented. By utilizing the prior low-rank property and temporal characteristic of the traffic matrix, the traffic matrix estimation in the coexistence of the outlier and structural anomaly is formulated as a Prior Information Guided Matrix Completion (PigMaC) problem, where outlier and structural anomalies are modeled by the L1-norm and L2,1-norm, respectively. Furthermore, by employing the multi-blocks ADMM and stochastic proximal gradient descent, a scalable parallel optimization algorithm PigMaC-ADMM-S is proposed to solve the PigMaC problem. To our knowledge, SETMADA is the first approach that can simultaneously estimate the missing traffic matrix and explicitly sift outlier and structural anomalies. Simulation results demonstrate that SETMADA achieves better estimation performance compared with the state-of-the-art algorithms. In addition, SETMADA provides an accurate localization of outlier and structural anomaly, which is the prerequisite for malfunction diagnosis in the large-scale networks.

[1]  Jin Cao,et al.  A Quasi-Likelihood Approach for Accurate Traffic Matrix Estimation in a High Speed Network , 2008, IEEE INFOCOM 2008 - The 27th Conference on Computer Communications.

[2]  Jennifer Rexford,et al.  Sensitivity of PCA for traffic anomaly detection , 2007, SIGMETRICS '07.

[3]  Martin May,et al.  Applying PCA for Traffic Anomaly Detection: Problems and Solutions , 2009, IEEE INFOCOM 2009.

[4]  Patrick L. Combettes,et al.  Signal Recovery by Proximal Forward-Backward Splitting , 2005, Multiscale Model. Simul..

[5]  Baolin Yin,et al.  Structural analysis of network traffic matrix via relaxed principal component pursuit , 2011, Comput. Networks.

[6]  Peng Zhang,et al.  A transform domain-based anomaly detection approach to network-wide traffic , 2014, J. Netw. Comput. Appl..

[7]  Jinfeng Yi,et al.  Stochastic Optimization for Kernel PCA , 2016, AAAI.

[8]  Michael Elad,et al.  From Sparse Solutions of Systems of Equations to Sparse Modeling of Signals and Images , 2009, SIAM Rev..

[9]  Morteza Mardani,et al.  Estimating Traffic and Anomaly Maps via Network Tomography , 2014, IEEE/ACM Transactions on Networking.

[10]  Lei Guo,et al.  Traffic matrix prediction and estimation based on deep learning in large-scale IP backbone networks , 2016, J. Netw. Comput. Appl..

[11]  Yong Yu,et al.  Robust Recovery of Subspace Structures by Low-Rank Representation , 2010, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[12]  Zhihan Lv,et al.  Spatio-Temporal Kronecker Compressive Sensing for Traffic Matrix Recovery , 2016, IEEE Access.

[13]  M. Brand,et al.  Fast low-rank modifications of the thin singular value decomposition , 2006 .

[14]  Liansheng Tan,et al.  Traffic matrix estimation: A neural network approach with extended input and expectation maximization iteration , 2016, J. Netw. Comput. Appl..

[15]  Lei Chen,et al.  Noise-tolerant localization from incomplete range measurements for wireless sensor networks , 2015, 2015 IEEE Conference on Computer Communications (INFOCOM).

[16]  Balachander Krishnamurthy,et al.  Flash crowds and denial of service attacks: characterization and implications for CDNs and web sites , 2002, WWW.

[17]  C. Berenstein,et al.  Network tomography , 2007 .

[18]  Gaogang Xie,et al.  Accurate recovery of Internet traffic data: A tensor completion approach , 2016, IEEE INFOCOM 2016 - The 35th Annual IEEE International Conference on Computer Communications.

[19]  Morteza Mardani,et al.  Robust network traffic estimation via sparsity and low rank , 2013, 2013 IEEE International Conference on Acoustics, Speech and Signal Processing.

[20]  Konstantina Papagiannaki,et al.  Structural analysis of network traffic flows , 2004, SIGMETRICS '04/Performance '04.

[21]  Walter Willinger,et al.  Spatio-Temporal Compressive Sensing and Internet Traffic Matrices (Extended Version) , 2012, IEEE/ACM Transactions on Networking.

[22]  Albert G. Greenberg,et al.  Fast accurate computation of large-scale IP traffic matrices from link loads , 2003, SIGMETRICS '03.

[23]  Zhixun Su,et al.  Linearized alternating direction method with parallel splitting and adaptive penalty for separable convex programs in machine learning , 2013, Machine Learning.

[24]  Paul Barford,et al.  A signal analysis of network traffic anomalies , 2002, IMW '02.

[25]  Tianguang Chu,et al.  Structure Regularized Traffic Monitoring for Traffic Matrix Estimation and Anomaly Detection by Link-Load Measurements , 2016, IEEE Transactions on Instrumentation and Measurement.

[26]  Mikkel Thorup,et al.  Traffic engineering with estimated traffic matrices , 2003, IMC '03.

[27]  Ling Huang,et al.  ANTIDOTE: understanding and defending against poisoning of anomaly detectors , 2009, IMC '09.

[28]  Yi Ma,et al.  Robust principal component analysis? , 2009, JACM.

[29]  Emmanuel J. Candès,et al.  Exact Matrix Completion via Convex Optimization , 2009, Found. Comput. Math..

[30]  Yin Zhang,et al.  Robust network compressive sensing , 2014, MobiCom.

[31]  Christophe Diot,et al.  Diagnosing network-wide traffic anomalies , 2004, SIGCOMM.

[32]  Emmanuel J. Candès,et al.  A Singular Value Thresholding Algorithm for Matrix Completion , 2008, SIAM J. Optim..

[33]  Bu Sung Lee Francis,et al.  Combining MIC feature selection and feature-based MSPCA for network traffic anomaly detection , 2016, 2016 Third International Conference on Digital Information Processing, Data Mining, and Wireless Communications (DIPDMWC).