A Framework for Automated Learning of Application Memory Usage Behaviour

Accurate estimation of the amount of memory required is an important step in discovering and selecting configurations for computational jobs in a grid environment. In order to mitigate the adverse impact of over and under estimation of memory requirements, job submitters need to carry out benchmarking in order to understand the memory usage behaviour of specific applications. An automated mechanism for learning the memory usage pattern can allow them to focus more on the results of the jobs. This paper presents mechanisms that automate the process of understanding the memory usage behaviour of jobs. The mechanism uses the provenance data from previously completed jobs. The proposed mechanism enables populating an information model describing the memory usage patterns of applications. The paper also presents techniques for making predictions on the memory usage while the development of the knowledge base is still being carried out. The utility of the mechanism is demonstrated by showing its increasing ability to predict the memory usage for incoming jobs. Also highlighted are the factors that effect the pace in which the system acquires knowledge about the memory usage behaviour.