Power Management of Extreme-Scale Networks with On/Off Links in Runtime Systems

Networks are among major power consumers in large-scale parallel systems. During execution of common parallel applications, a sizeable fraction of the links in the high-radix interconnects are either never used or are underutilized. We propose a runtime system based adaptive approach to turn off unused links, which has various advantages over the previously proposed hardware and compiler based approaches. We discuss why the runtime system is the best system component to accomplish this task, and test the effectiveness of our approach using real applications (including NAMD, MILC), and application benchmarks (including NAS Parallel Benchmarks, Stencil). These codes are simulated on representative topologies such as 6-D Torus and multilevel directly connected network (similar to IBM PERCS in Power 775 and Dragonfly in Cray Aries). For common applications with near-neighbor communication pattern, our approach can save up to 20% of total machine's power and energy, without any performance penalty.

[1]  Laxmikant V. Kalé,et al.  Quantifying Network Contention on Large Parallel Machines , 2009, Parallel Process. Lett..

[2]  Philip Heidelberger,et al.  The IBM Blue Gene/Q interconnection network and message unit , 2011, 2011 International Conference for High Performance Computing, Networking, Storage and Analysis (SC).

[3]  C. DeTar,et al.  Scaling tests of the improved Kogut-Susskind quark action , 1999, hep-lat/9912018.

[4]  Laxmikant V. Kalé,et al.  NAMD (NAnoscale Molecular Dynamics) , 2011, Encyclopedia of Parallel Computing.

[5]  Laxmikant V. Kalé,et al.  Periodic hierarchical load balancing for large supercomputers , 2011, Int. J. High Perform. Comput. Appl..

[6]  Holger Fröning,et al.  NAnoscale Molecular Dynamics (NAMD) , 2011, Encyclopedia of Parallel Computing.

[7]  Li Shang,et al.  Dynamic voltage scaling with links for power optimization of interconnection networks , 2003, The Ninth International Symposium on High-Performance Computer Architecture, 2003. HPCA-9 2003. Proceedings..

[8]  Li-Shiuan Peh,et al.  Software-directed power-aware interconnection networks , 2007, ACM Trans. Archit. Code Optim..

[9]  Pedro López,et al.  Dynamic power saving in fat-tree interconnection networks using on/off links , 2006, Proceedings 20th IEEE International Parallel & Distributed Processing Symposium.

[10]  Laxmikant V. Kalé,et al.  BigSim: a parallel simulator for performance prediction of extremely large parallel machines , 2004, 18th International Parallel and Distributed Processing Symposium, 2004. Proceedings..

[11]  Laxmikant V. Kalé,et al.  Avoiding hot-spots on two-level direct networks , 2011, 2011 International Conference for High Performance Computing, Networking, Storage and Analysis (SC).

[12]  Hong Liu,et al.  Energy proportional datacenter networks , 2010, ISCA.

[13]  Sameer Kumar,et al.  Acceleration of an Asynchronous Message Driven Programming Paradigm on IBM Blue Gene/Q , 2013, 2013 IEEE 27th International Symposium on Parallel and Distributed Processing.

[14]  Sujata Banerjee,et al.  Energy Aware Network Operations , 2009, IEEE INFOCOM Workshops 2009.

[15]  Laxmikant V. Kalé,et al.  Toward Runtime Power Management of Exascale Networks by on/off Control of Links , 2013, 2013 IEEE International Symposium on Parallel & Distributed Processing, Workshops and Phd Forum.

[16]  Gilbert Hendry.  Decreasing Network Power with on-off Links Informed by Scientific Applications , 2013, 2013 IEEE International Symposium on Parallel & Distributed Processing, Workshops and Phd Forum.

[17]  Courtenay T. Vaughan,et al.  Energy based performance tuning for large scale high performance computing systems , 2012, HiPC 2012.

[18]  B GibbonsPhillip.  ACM transactions on parallel computing , 2014 .

[19]  Josep Torrellas,et al.  Comparing the power and performance of Intel's SCC to state-of-the-art CPUs and GPUs , 2012, 2012 IEEE International Symposium on Performance Analysis of Systems & Software.

[20]  Aaron Karl Becker,et al.  Compiler support for productive message-driven parallel programming , 2012 .

[21]  Laxmikant V. Kale,et al.  Charm++ and AMPI: Adaptive Runtime Strategies via Migratable Objects , 2009 .

[22]  Atul K. Jain,et al.  Modeling the effects of two different land cover change data sets on the carbon stocks of plants and soils in concert with CO2 and climate change , 2005 .

[23]  Laxmikant V. Kalé,et al.  "Cool" Load Balancing for High Performance Computing Data Centers , 2012, IEEE Trans. Computers.

[24]  Jian Li,et al.  Power shifting in Thrifty Interconnection Network , 2011, 2011 IEEE 17th International Symposium on High Performance Computer Architecture.

[25]  Laxmikant V. Kalé,et al.  Optimizing communication for Charm++ applications by reducing network contention , 2011, Concurr. Comput. Pract. Exp..

[26]  Sujata Banerjee,et al.  ElasticTree: Saving Energy in Data Center Networks , 2010, NSDI.

[27]  David H. Bailey,et al.  NAS parallel benchmark results , 1992, Proceedings Supercomputing '92.

[28]  Abhinav Vishnu,et al.  Comparing the Performance of Blue Gene/Q with Leading Cray XE6 and InfiniBand Systems , 2012, 2012 IEEE 18th International Conference on Parallel and Distributed Systems.

[29]  Mary Jane Irwin,et al.  Link Shutdown Opportunities During Collective Communications in 3-D Torus Nets , 2007, 2007 IEEE International Parallel and Distributed Processing Symposium.

[30]  William J. Dally,et al.  Technology-Driven, Highly-Scalable Dragonfly Topology , 2008, 2008 International Symposium on Computer Architecture.

[31]  Mike Higgins,et al.  Cray Cascade: A scalable HPC system based on a Dragonfly network , 2012, 2012 International Conference for High Performance Computing, Networking, Storage and Analysis.

[32]  Laxmikant V. Kalé,et al.  Simulation-Based Performance Analysis and Tuning for a Two-Level Directly Connected System , 2011, 2011 IEEE 17th International Conference on Parallel and Distributed Systems.

[33]  Li-Shiuan Peh,et al.  Exploring the Design Space of Self-Regulating Power-Aware On/Off Interconnection Networks , 2007, IEEE Transactions on Parallel and Distributed Systems.

[34]  Tomohiro Inoue,et al.  The Tofu Interconnect , 2011, 2011 IEEE 19th Annual Symposium on High Performance Interconnects.

[35]  Laxmikant V. Kalé,et al.  Scalable Algorithms for Distributed-Memory Adaptive Mesh Refinement , 2012, 2012 IEEE 24th International Symposium on Computer Architecture and High Performance Computing.

[36]  Torsten Hoefler,et al.  The PERCS High-Performance Interconnect , 2010, 2010 18th IEEE Symposium on High Performance Interconnects.