Application dependency discovery using matrix factorization

Driven by the large-scale growth of applications deployment in data centers and complicated interactions between service components, automated application dependency discovery becomes essential to daily system management and operation. In this paper, we present ADD, which extracts dependency paths for each application by decomposing the application-layer connectivity graph inferred from passive network monitoring data. ADD utilizes a series of statistical techniques and is based on the combination of global observation of application traffic matrix in the data center and local observation of traffic volumes at small time scales on each server. Compared to existing approaches, ADD is especially effective in the presence of overlapping and multi-hop applications and resilient to data loss and estimation errors.

[1]  Paramvir Bahl,et al.  Towards highly reliable enterprise network services via inference of multi-level dependencies , 2007, SIGCOMM '07.

[2]  Chun Zhang,et al.  vPath: Precise Discovery of Request Processing Paths from Black-Box Observations of Thread and Network Activities , 2009, USENIX Annual Technical Conference.

[3]  Richard Mortier,et al.  Using Magpie for Request Extraction and Workload Modelling , 2004, OSDI.

[4]  Spyros G. Denazis,et al.  Dependency Detection Using a Fuzzy Engine , 2007, DSOM.

[5]  Stephen P. Boyd,et al.  Rank minimization and applications in system theory , 2004, Proceedings of the 2004 American Control Conference.

[6]  Haifeng Chen,et al.  Efficient and Scalable Algorithms for Inferring Likely Invariants in Distributed Systems , 2007, IEEE Transactions on Knowledge and Data Engineering.

[7]  René Vidal,et al.  Motion Segmentation with Missing Data Using PowerFactorization and GPCA , 2004, CVPR.

[8]  Paramvir Bahl,et al.  Discovering Dependencies for Network Management , 2006, HotNets.

[9]  H. Sebastian Seung,et al.  Algorithms for Non-negative Matrix Factorization , 2000, NIPS.

[10]  Ranveer Chandra,et al.  What's going on?: learning communication rules in edge networks , 2008, SIGCOMM '08.

[11]  Lawrence K. Saul,et al.  Modeling distances in large-scale networks by matrix factorization , 2004, IMC '04.

[12]  Xu Chen,et al.  Automating Network Application Dependency Discovery: Experiences, Limitations, and New Solutions , 2008, OSDI.

[13]  R. Hartley,et al.  PowerFactorization : 3D reconstruction with missing or uncertain data , 2003 .