A Year in the Life of a Parallel File System

I/O performance is a critical aspect of data-intensive scientific computing. We seek to advance the state of the practice in understanding and diagnosing I/O performance issues through investigation of a comprehensive I/O performance data set that captures a full year of production storage activity at two leadership-scale computing facilities. We demonstrate techniques to identify regions of interest, perform focused investigations of both long-term trends and transient anomalies, and uncover the contributing factors that lead to performance fluctuation. We find that a year in the life of a parallel file system is comprised of distinct regions of long-term performance variation in addition to short-term performance transients. We demonstrate how systematic identification of these performance regions, combined with comprehensive analysis, allows us to isolate the factors contributing to different performance maladies at different time scales. From this, we present specific lessons learned and important considerations for HPC storage practitioners.

[1]  Andy B. Yoo,et al.  Approved for Public Release; Further Dissemination Unlimited X-ray Pulse Compression Using Strained Crystals X-ray Pulse Compression Using Strained Crystals , 2002 .

[2]  Julian M. Kunkel,et al.  The SIOX Architecture - Coupling Automatic Monitoring and Optimization of Parallel I/O , 2014, ISC.

[3]  Hal Finkel,et al.  The Universe at extreme scale: Multi-petaflop sky simulation on the BG/Q , 2012, 2012 International Conference for High Performance Computing, Networking, Storage and Analysis.

[4]  Joseph ’Joshi.  Nuclear Meltdown? Assessing the impact of the Meltdown/Spectre bug at Los Alamos National Laboratory LA-UR-18-24290 , 2018 .

[5]  Scott Klasky,et al.  Characterizing output bottlenecks in a supercomputer , 2012, 2012 International Conference for High Performance Computing, Networking, Storage and Analysis.

[6]  Christian Engelmann,et al.  Big Data Meets HPC Log Analytics: Scalable Approach to Understanding Systems at Extreme Scale , 2017, 2017 IEEE International Conference on Cluster Computing (CLUSTER).

[7]  Devesh Tiwari,et al.  GUIDE: A Scalable Information Directory Service to Collect, Federate, and Analyze Logs for Operational Insights into a Leadership HPC Facility , 2017, SC17: International Conference for High Performance Computing, Networking, Storage and Analysis.

[8]  Robert Latham,et al.  Machine Learning Based Parallel I/O Predictive Modeling: A Case Study on Lustre File Systems , 2018, ISC.

[9]  Saurabh Gupta,et al.  Best Practices and Lessons Learned from Deploying and Operating Large-Scale Data-Centric Parallel File Systems , 2014, SC14: International Conference for High Performance Computing, Networking, Storage and Analysis.

[10]  Kevin Harms,et al.  Scalable Parallel I/O on a Blue Gene/Q Supercomputer Using Compression, Topology-Aware Data Aggregation, and Subfiling , 2014, 2014 22nd Euromicro International Conference on Parallel, Distributed, and Network-Based Processing.

[11]  Karsten Schwan,et al.  Managing Variability in the IO Performance of Petascale Storage Systems , 2010, 2010 ACM/IEEE International Conference for High Performance Computing, Networking, Storage and Analysis.

[12]  Donald Beaver,et al.  Dapper, a Large-Scale Distributed Systems Tracing Infrastructure , 2010 .

[13]  Franck Cappello,et al.  LOGAIDER: A Tool for Mining Potential Correlations of HPC Log Events , 2017, 2017 17th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing (CCGRID).

[14]  Marianne Winslett,et al.  A Multiplatform Study of I/O Behavior on Petascale Supercomputers , 2015, HPDC.

[15]  David Power,et al.  The profitability of moving average trading rules in South Asian stock markets , 2001 .

[16]  Robert B. Ross,et al.  Fail-Slow at Scale , 2018, ACM Trans. Storage.

[17]  Thomas W. Tucker,et al.  The Lightweight Distributed Metric Service: A Scalable Infrastructure for Continuous Monitoring of Large Scale Computing Systems and Applications , 2014, SC14: International Conference for High Performance Computing, Networking, Storage and Analysis.

[18]  F. E. James.  Monthly Moving Averages—An Effective Investment Tool? , 1968, Journal of Financial and Quantitative Analysis.

[19]  Surendra Byna,et al.  BD-CATS: big data clustering at trillion particle scale , 2015, SC15: International Conference for High Performance Computing, Networking, Storage and Analysis.

[20]  Stephen A. Jarvis,et al.  Parallel File System Analysis Through Application I/O Tracing , 2013, Comput. J..

[21]  Kevin Harms,et al.  TOKIO on ClusterStor: Connecting Standard Tools to Enable Holistic I/O Performance Analysis , 2018 .

[22]  Robert Latham,et al.  Performance Evaluation of Darshan 3.0.0 on the Cray XC30 , 2016 .

[23]  Robert Latham,et al.  24/7 Characterization of petascale I/O workloads , 2009, 2009 IEEE International Conference on Cluster Computing and Workshops.

[24]  Arnold D. Well,et al.  Many Faces of the Correlation Coefficient , 1997 .

[25]  Kevin Harms,et al.  UMAMI: a recipe for generating meaningful metrics through holistic I/O performance analysis , 2017, PDSW-DISCS@SC.

[26]  Robert B. Ross,et al.  On the Root Causes of Cross-Application I/O Interference in HPC Storage Systems , 2016, 2016 IEEE International Parallel and Distributed Processing Symposium (IPDPS).

[27]  Robert Latham,et al.  Understanding and improving computational science storage access through continuous characterization , 2011, MSST.