Power usage of production supercomputers and production workloads

Power is becoming an increasingly important concern for large supercomputer centers. However, to date, there have been a dearth of studies of power usage ‘in the wild’—on production supercomputers running production workloads. In this paper, we present the initial results of a project to characterize the power usage of the three Top500 supercomputers at Los Alamos National Laboratory: Cielo, Roadrunner, and Luna (#15, #19, and #47, respectively, on the June 2012 Top500 list). Power measurements taken both at the switchboard level and within the compute racks are presented and discussed. Some noteworthy results of this study are that (1) variability in power consumption differs across architectures, even when running a similar workload and (2) Los Alamos National Laboratory's scientific workload draws, on average, only 70–75% of LINPACK power and only 40–55% of nameplate power, implying that power capping may enable a substantial reduction in power and cooling infrastructure while impacting comparatively few applications. Copyright © 2013 John Wiley & Sons, Ltd.

[1]  Bianca Schroeder,et al.  A Large-Scale Study of Failures in High-Performance Computing Systems , 2006, IEEE Transactions on Dependable and Secure Computing.

[2]  G. Lafontant,et al.  Packaging the Cell Broadband Engine microprocessor for supercomputer applications , 2008, 2008 58th Electronic Components and Technology Conference.

[3]  Robert Weaver,et al.  The RAGE radiation-hydrodynamic code , 2008 .

[4]  Steve Greenberg,et al.  Best Practices for Data Centers: Lessons Learned from Benchmarking 22 Data Centers , 2006 .

[5]  Wolf-Dietrich Weber,et al.  Power provisioning for a warehouse-sized computer , 2007, ISCA '07.

[6]  Courtenay T. Vaughan,et al.  Investigating the Impact of the Cielo Cray XE6 Architecture on Scientific Application Codes , 2011, 2011 IEEE International Symposium on Parallel and Distributed Processing Workshops and Phd Forum.

[7]  P. J. Green,et al.  Density Estimation for Statistics and Data Analysis , 1987 .

[8]  Venkatesh Pallipadi,et al.  The Ondemand Governor Past, Present, and Future , 2010 .

[9]  Courtenay T. Vaughan,et al.  Energy based performance tuning for large scale high performance computing systems , 2012, HiPC 2012.

[10]  Bruce Hendrickson,et al.  The Torus-Wrap Mapping for Dense Matrix Calculations on Massively Parallel Computers , 1994, SIAM J. Sci. Comput..

[11]  Jack J. Dongarra,et al.  The LINPACK Benchmark: past, present and future , 2003, Concurr. Comput. Pract. Exp..

[12]  Pawel Gepner,et al.  Early Performance Evaluation of New Six-Core Intel® Xeon® 5600 Family Processors for HPC , 2010, 2010 Ninth International Symposium on Parallel and Distributed Computing.

[13]  Michael Lang,et al.  Entering the petaflop era: The architecture and performance of Roadrunner , 2008, 2008 SC - International Conference for High Performance Computing, Networking, Storage and Analysis.

[14]  Wolfgang Frings,et al.  Measuring power consumption on IBM Blue Gene/P , 2011, Computer Science - Research and Development.

[15]  H. Peter Hofstee,et al.  Introduction to the Cell multiprocessor , 2005, IBM J. Res. Dev..

[16]  George L.-T. Chiu,et al.  Overview of the Blue Gene/L system architecture , 2005, IBM J. Res. Dev..

[17]  Wu-chun Feng,et al.  The Green500 List: Year two , 2010, 2010 IEEE International Symposium on Parallel & Distributed Processing, Workshops and Phd Forum (IPDPSW).

[18]  F. Petrini,et al.  The Case of the Missing Supercomputer Performance: Achieving Optimal Performance on the 8,192 Processors of ASCI Q , 2003, ACM/IEEE SC 2003 Conference (SC'03).

[19]  Christian Belady,et al.  GREEN GRID DATA CENTER POWER EFFICIENCY METRICS: PUE AND DCIE , 2008 .

[20]  Hans Werner Meuer,et al.  Top500 Supercomputer Sites , 1997 .

[21]  Timothy C. Germann,et al.  369 Tflop/s molecular dynamics simulations on the Roadrunner general-purpose heterogeneous supercomputer , 2008, 2008 SC - International Conference for High Performance Computing, Networking, Storage and Analysis.

[22]  J. Wellner,et al.  Empirical Processes with Applications to Statistics , 2009 .

[23]  John Shalf,et al.  The International Exascale Software Project roadmap , 2011, Int. J. High Perform. Comput. Appl..

[24]  Rong Ge,et al.  Green Supercomputing Comes of Age , 2008, IT Professional.

[25]  Larry Kaplan,et al.  The Gemini System Interconnect , 2010, 2010 18th IEEE Symposium on High Performance Interconnects.

[26]  G. Casella.  An Introduction to Empirical Bayes Data Analysis , 1985 .