Optimization of infiniband for scientific applications

The achievable performance on Infiniband networks is governed by the latencies and bandwidths of communication channels as well as by contention within the network. Currently Infiniband statically routes messages and thus do not take into account dynamic loading of the channels. By interrogating the network routing tables we quantify the contention that occurs for a number of communication patterns using a large-scale (1024 processor) system. Empirical data confirms our contention calculation almost exactly. Custom routing tables are defined that provide both optimum and worst-case performance for a large- range of communication patterns. Performance differences can be as large as 12x (from optimum to worst-case). Two large-scale applications show a runtime improvement of between 10-20% and up to 40% improvement in just their communication time when using optimized routing tables. The approach taken is applicable to many Infiniband systems, and we expect the performance improvements to be even greater on larger-scale systems.

[1]  Scott Pakin.  CONCEPTUAL: a network correctness and performance testing language , 2004, 18th International Parallel and Distributed Processing Symposium, 2004. Proceedings..

[2]  G. Johnson,et al.  A Performance Comparison Through Benchmarking and Modeling of Three Leading Supercomputers: Blue Gene/L, Red Storm, and Purple , 2006, ACM/IEEE SC 2006 Conference (SC'06).

[3]  Z. Ding,et al.  Level-wise Scheduling Algorithm for Fat Tree Interconnection Networks , 2006, ACM/IEEE SC 2006 Conference (SC'06).

[4]  Fabrizio Petrini,et al.  Predictive Performance and Scalability Modeling of a Large-Scale Application , 2001, ACM/IEEE SC 2001 Conference (SC'01).

[5]  David F. Heidel,et al.  An Overview of the BlueGene/L Supercomputer , 2002, ACM/IEEE SC 2002 Conference (SC'02).

[6]  Leonid Oliker,et al.  Reconfigurable hybrid interconnection for static and dynamic scientific applications , 2007, CF '07.

[7]  Rami G. Melhem,et al.  On the Feasibility of Optical Circuit Switching for High Performance Computing Systems , 2005, ACM/IEEE SC 2005 Conference (SC'05).

[8]  Darren J. Kerbyson.  A look at application performance sensitivity to the bandwidth and latency of InfiniBand networks , 2006, Proceedings 20th IEEE International Parallel & Distributed Processing Symposium.

[9]  Darren J. Kerbyson,et al.  Optimized InfiniBand TM fat-tree routing for shift all-to-all communication patterns , 2010, ISC 2010.

[10]  J. Nieplocha,et al.  QSNET/sup II/: defining high-performance network design , 2005, IEEE Micro.

[11]  R. Baker.  A Block Adaptive Mesh Refinement Algorithm for the Neutral Particle Transport Equation , 2002 .

[12]  Darren J. Kerbyson,et al.  Automatic Identification of Application Communication Patterns via Templates , 2005, ISCA PDCS.

[13]  Michael Lang,et al.  Optimized InfiniBandTM fat‐tree routing for shift all‐to‐all communication patterns , 2010, Concurr. Comput. Pract. Exp..

[14]  Ingrid Verbauwhede,et al.  High-throughput programmable cryptocoprocessor , 2004, IEEE Micro.