Automated clustering of VMs for scalable cloud monitoring and management

The size of modern datacenters supporting cloud computing represents a major challenge in terms of monitoring and management of system resources. Available solutions typically consider every Virtual Machine (VM) as a black box each with independent characteristics and face scalability issues by reducing the number of monitoring resource samples, considering in most cases only average CPU utilization of VMs sampled at a very coarse time granularity. We claim that better management without compromising scalability could be achieved by clustering together VMs that show similar behavior in terms of resource utilization. In this paper we propose an automated methodology to cluster VMs depending on the utilization of their resources, assuming no knowledge of the services executed on them. The methodology considers several VM resources, both system-and network-related, and exploits the correlation between the resource demand to cluster together similar VMs. We apply the proposed methodology to a case study with data coming from an enterprise datacenter to evaluate the accuracy of VMs clustering and to estimate the reduction in the amount of data collected. The automatic clustering achieved through our approach may simplify the monitoring requirements and help administrators to take decisions on the management of the resources in a cloud computing datacenter.

[1]  Jerome A. Rolia,et al.  Resource pool management: Reactive versus proactive or let's be friends , 2009, Comput. Networks.

[2]  S. M. Ibrahim Lavlu,et al.  Cacti 0.8 Network Monitoring , 2009 .

[3]  Alexander Stage,et al.  Decision support for virtual machine reassignments in enterprise data centers , 2010, 2010 IEEE/IFIP Network Operations and Management Symposium Workshops.

[4]  Steven Shiau,et al.  A Power-Aware Cloud Architecture with Smart Metering , 2010, 2010 39th International Conference on Parallel Processing Workshops.

[5]  Michele Colajanni,et al.  A Software Architecture for the Analysis of Large Sets of Data Streams in Cloud Infrastructures , 2011, 2011 IEEE 11th International Conference on Computer and Information Technology.

[6]  Arun Venkataramani,et al.  Black-box and Gray-box Strategies for Virtual Machine Migration , 2007, NSDI.

[7]  Ruay-Shiung Chang,et al.  A new mechanism for resource monitoring in Grid computing , 2009, Future Gener. Comput. Syst..

[8]  Sriram Sankar,et al.  Measuring Database Performance in Online Services: A Trace-Based Approach , 2009, TPCTC.

[9]  Neil D. Lawrence,et al.  Automatic Determination of the Number of Clusters Using Spectral Algorithms , 2005, 2005 IEEE Workshop on Machine Learning for Signal Processing.

[10]  Sameh Elnikety,et al.  Performance Comparison of Middleware Architectures for Generating Dynamic Web Content , 2003, Middleware.

[11]  Alexander Stage,et al.  Filtering multivariate workload non-conformance in shared IT-infrastructures , 2011, 12th IFIP/IEEE International Symposium on Integrated Network Management (IM 2011) and Workshops.

[12]  Prashant J. Shenoy,et al.  Predico: A System for What-if Analysis in Complex Data Center Applications , 2011, Middleware.

[13]  Anil K. Jain.  Data clustering: 50 years beyond K-means , 2008, Pattern Recognit. Lett..

[14]  Rajkumar Buyya,et al.  Adaptive threshold-based approach for energy-efficient consolidation of virtual machines in cloud data centers , 2010, MGC '10.

[15]  Yong,et al.  Controlling Scale Sensor Networks Data Quality in the Ganglia Grid Monitoring Tool , 2010 .

[16]  Parijat Dube,et al.  Exploiting Resource Usage Patterns for Better Utilization Prediction , 2011, 2011 31st International Conference on Distributed Computing Systems Workshops.