CloudScout: A Non-Intrusive Approach to Service Dependency Discovery

Nowadays, numerous enterprises are migrating their applications into cloud computing environments. Typically, the applications are composed of several dependent service components that span many hosts and network devices. In light of this, exploring the dependency between service components can be beneficial for achieving fast network application response time. Moreover, it is significant to consolidate service components according to resource constraints, service dependency, and network structure. However, it is a tedious task to discover the dependency among service components without expert knowledge of the running application. In this paper, we propose CloudScout, a non-intrusive approach that is capable of automatically discovering dependent service components. CloudScout analyzes the correlation among service components based on the time-series information from system monitoring logs. We address two key challenges in CloudScout: service distance calculation and dependent service clustering. We conduct experiments on five applications with 290 service components that span 20 physical hosts across two data centers. The experimental results demonstrate that CloudScout can successfully discover the dependency among service components and facilitate reducing the network latency of network applications and distributed applications.

[1]  Jianwei Yin,et al.  Distance-aware virtual cluster performance optimization: A hadoop case study , 2013, 2013 IEEE International Conference on Cluster Computing (CLUSTER).

[2]  Sushil Jajodia,et al.  NSDMiner: Automated discovery of Network Service Dependencies , 2012, 2012 Proceedings IEEE INFOCOM.

[3]  Chita R. Das,et al.  Modeling and synthesizing task placement constraints in Google compute clusters , 2011, SoCC.

[4]  Sujata Banerjee,et al.  Application-driven bandwidth guarantees in datacenters , 2014, SIGCOMM.

[5]  Karsten Schwan,et al.  Look Who's Talking: Discovering Dependencies between Virtual Machines Using CPU Utilization , 2010, HotCloud.

[6]  Rui Xu,et al.  Survey of clustering algorithms , 2005, IEEE Transactions on Neural Networks.

[7]  Prashant J. Shenoy,et al.  Analytical modeling for what-if analysis in complex cloud computing applications , 2013, PERV.

[8]  Chun Zhang,et al.  vPath: Precise Discovery of Request Processing Paths from Black-Box Observations of Thread and Network Activities , 2009, USENIX Annual Technical Conference.

[9]  Mike P. Papazoglou,et al.  Service oriented architectures: approaches, technologies and research issues , 2007, The VLDB Journal.

[10]  Marcos K. Aguilera,et al.  Performance debugging for distributed systems of black boxes , 2003, SOSP '03.

[11]  Paramvir Bahl,et al.  Towards highly reliable enterprise network services via inference of multi-level dependencies , 2007, SIGCOMM '07.

[12]  Marcos K. Aguilera,et al.  WAP5: black-box performance debugging for wide-area systems , 2006, WWW '06.

[13]  Úlfar Erlingsson,et al.  Fay: extensible distributed tracing from kernels to clusters , 2011, SOSP '11.

[14]  Calton Pu,et al.  Who Is Your Neighbor: Net I/O Performance Interference in Virtualized Clouds , 2013, IEEE Transactions on Services Computing.

[15]  Thanasis Loukopoulos,et al.  Application-Aware Workload Consolidation to Minimize Both Energy Consumption and Network Load in Cloud Environments , 2013, 2013 42nd International Conference on Parallel Processing.

[16]  Christos Faloutsos,et al.  RainMon: an integrated approach to mining bursty timeseries monitoring data , 2012, KDD.

[17]  Eric A. Brewer,et al.  Pinpoint: problem determination in large, dynamic Internet services , 2002, Proceedings International Conference on Dependable Systems and Networks.

[18]  Karsten Schwan,et al.  Net-cohort: detecting and managing VM ensembles in virtualized data centers , 2012, ICAC '12.

[19]  Jianwei Yin,et al.  Workload Classification Model for Specializing Virtual Machine Operating System , 2013, 2013 IEEE Sixth International Conference on Cloud Computing.

[20]  William H. Sanders,et al.  Blackbox prediction of the impact of DVFS on end-to-end performance of multitier systems , 2010, PERV.

[21]  Asser N. Tantawi,et al.  An analytical model for multi-tier internet services and its applications , 2005, SIGMETRICS '05.

[22]  Richard Mortier,et al.  Using Magpie for Request Extraction and Workload Modelling , 2004, OSDI.

[23]  Antonio Corradi,et al.  A Stable Network-Aware VM Placement for Cloud Systems , 2012, 2012 12th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing (ccgrid 2012).

[24]  Xu Chen,et al.  Automating Network Application Dependency Discovery: Experiences, Limitations, and New Solutions , 2008, OSDI.

[25]  Jun Yan,et al.  A Network-aware Virtual Machine Placement and Migration Approach in Cloud Computing , 2010, 2010 Ninth International Conference on Grid and Cloud Computing.

[26]  Fung Po Tso,et al.  Scalable Traffic-Aware Virtual Machine Management for Cloud Data Centers , 2014, 2014 IEEE 34th International Conference on Distributed Computing Systems.

[27]  Qiang Fu,et al.  Mining dependency in distributed systems through unstructured logs analysis , 2010, OPSR.