Analyzing IO Usage Patterns of User Jobs to Improve Overall HPC System Efficiency

This work looks at analyzing I/O traffic of users’ jobs on a HPC machine for a period of time. Monitoring tools are collecting the data in a continuous basis on the HPC system. We looked at aggregate I/O data usage patterns of users’ jobs on the system both on the parallel shared Lustre file system and the node-local SSDs. Data mining tools are then applied to analyze the I/O usage pattern data in an attempt to tie the data to particular codes that produced those I/O behaviors from users’ jobs.

[1]  Hans-Peter Kriegel,et al.  A Density-Based Algorithm for Discovering Clusters in Large Spatial Databases with Noise , 1996, KDD.

[2]  Hans-Peter Kriegel,et al.  Density‐based clustering , 2011, WIREs Data Mining Knowl. Discov..

[3]  Gaël Varoquaux,et al.  Scikit-learn: Machine Learning in Python , 2011, J. Mach. Learn. Res..

[4]  Plamen Angelov,et al.  Data density based clustering , 2014, 2014 14th UK Workshop on Computational Intelligence (UKCI).