Swap-And-Randomize: A Method for Building Low-Latency HPC Interconnects

Random network topologies have been proposed to create low-diameter, low-latency interconnection networks in large-scale computing systems. However, these topologies are difficult to deploy in practice, especially when re-designing existing systems, because they lead to increased total cable length and cable packaging complexity. In this work we propose a new method for creating random topologies without increasing cable length: randomly swap link endpoints in a non-random topology that is already deployed across several cabinets in a machine room. We quantitatively evaluate topologies created in this manner using both graph analysis and cycle-accurate network simulation, including comparisons with non-random topologies and previously-proposed random topologies.

[1]  William J. Dally,et al.  Technology-Driven, Highly-Scalable Dragonfly Topology , 2008, 2008 International Symposium on Computer Architecture.

[2]  Dhiraj K. Pradhan,et al.  The Hyper-deBruijn Networks: Scalable Versatile Architecture , 1993, IEEE Trans. Parallel Distributed Syst..

[3]  J. Sirán,et al.  Moore Graphs and Beyond: A survey of the Degree/Diameter Problem , 2013 .

[4]  Federico Silla,et al.  High-Performance Routing in Networks of Workstations with Irregular Topology , 2000, IEEE Trans. Parallel Distributed Syst..

[5]  Sheldon B. Akers,et al.  The Star Graph: An Attractive Alternative to the n-Cube , 1994, ICPP.

[6]  William J. Dally,et al.  Principles and Practices of Interconnection Networks , 2004 .

[7]  Henri Casanova,et al.  Layout-conscious random topologies for HPC off-chip interconnects , 2013, 2013 IEEE 19th International Symposium on High Performance Computer Architecture (HPCA).

[8]  Béla Bollobás,et al.  The Diameter of a Cycle Plus a Random Matching , 1988, SIAM J. Discret. Math..

[9]  David Connolly An improved annealing scheme for the QAP , 1990 .

[10]  Kai Hwang,et al.  Hypernet: A Communication-Efficient Architecture for Constructing Massively Parallel Computers , 1987, IEEE Transactions on Computers.

[11]  Emin Gün Sirer,et al.  Small-world datacenters , 2011, SoCC.

[12]  Duncan J. Watts,et al.  Collective dynamics of ‘small-world’ networks , 1998, Nature.

[13]  D. Glickenstein Small world phenomenon and decentralized search , 2004 .

[14]  Henri Casanova,et al.  A case for random shortcut topologies for HPC interconnects , 2012, 2012 39th Annual International Symposium on Computer Architecture (ISCA).

[15]  William J. Dally,et al.  Flattened butterfly: a cost-efficient topology for high-radix networks , 2007, ISCA '07.

[16]  Toshiyuki Shimizu,et al.  Tofu: A 6D Mesh/Torus Interconnect for Exascale Computers , 2009, Computer.

[17]  Henri Casanova,et al.  Cabinet Layout Optimization of Supercomputer Topologies for Shorter Cable Length , 2012, 2012 13th International Conference on Parallel and Distributed Computing, Applications and Technologies.

[18]  Shahram Latifi,et al.  Properties and Performance of Folded Hypercubes , 1991, IEEE Trans. Parallel Distributed Syst..

[19]  Kemal Efe A Variation on the Hypercube with Lower Diameter , 1991, IEEE Trans. Computers.

[20]  José Duato,et al.  994 International Conference on Parallel Processing a Necessary and Sufficient Condition for Deadlock-free Adaptive Routing in Wormhole Networks , 2022 .

[21]  Nian-Feng Tzeng,et al.  Enhanced Hypercubes , 1991, IEEE Trans. Computers.

[22]  Magdy A. Bayoumi,et al.  The Hierarchical Hypercube: A New Interconnection Topology for Massively Parallel Systems , 1994, IEEE Trans. Parallel Distributed Syst..

[23]  Gerard V. Kopcsay,et al.  Packaging the Blue Gene/L supercomputer , 2005, IBM J. Res. Dev..

[24]  Ankit Singla,et al.  Jellyfish: Networking Data Centers Randomly , 2011, NSDI.

[25]  Dhiraj K. Pradhan,et al.  The De Bruijn Multiprocessor Network: A Versatile Parallel Processing and Sorting Network for VLSI , 1989, IEEE Trans. Computers.

[26]  Jung Ho Ahn,et al.  HyperX: topology, routing, and packaging of efficient large-scale networks , 2009, Proceedings of the Conference on High Performance Computing Networking, Storage and Analysis.

[27]  William J. Dally,et al.  Express Cubes: Improving the Performance of k-Ary n-Cube Interconnection Networks , 1989, IEEE Trans. Computers.