Slim Fly: A Cost Effective Low-Diameter Network Topology

We introduce a high-performance cost-effective network topology called Slim Fly that approaches the theoretically optimal network diameter. Slim Fly is based on graphs that approximate the solution to the degree-diameter problem. We analyze Slim Fly and compare it to both traditional and state-of the-art networks. Our analysis shows that Slim Fly has significant advantages over other topologies in latency, bandwidth, resiliency, cost, and power consumption. Finally, we propose deadlock-free routing schemes and physical layouts for large computing centres as well as a detailed cost and power model. Slim Fly enables constructing cost effective and highly resilient data enter and HPC networks that offer low latency and high bandwidth under different HPC workloads such as stencil or graph computations.

[1]  William J. Dally,et al.  Technology-Driven, Highly-Scalable Dragonfly Topology , 2008, 2008 International Symposium on Computer Architecture.

[2]  Henri Casanova,et al.  A case for random shortcut topologies for HPC interconnects , 2012, 2012 39th Annual International Symposium on Computer Architecture (ISCA).

[3]  William J. Dally,et al.  Flattened butterfly: a cost-efficient topology for high-radix networks , 2007, ISCA '07.

[4]  Mike Higgins,et al.  Cray Cascade: A scalable HPC system based on a Dragonfly network , 2012, 2012 International Conference for High Performance Computing, Networking, Storage and Analysis.

[5]  Torsten Hoefler,et al.  Accelerating Irregular Computations with Hardware Transactional Memory and Active Messages , 2015, HPDC.

[6]  Nan Jiang,et al.  A detailed and flexible cycle-accurate Network-on-Chip simulator , 2013, 2013 IEEE International Symposium on Performance Analysis of Systems and Software (ISPASS).

[7]  Ratko V. Tomic Network Throughput Optimization via Error Correcting Codes , 2013, ArXiv.

[8]  Ankit Singla,et al.  Jellyfish: Networking Data Centers Randomly , 2011, NSDI.

[9]  Geng Lin,et al.  Fault-tolerant circuit-switching networks , 1992, SPAA '92.

[10]  Nan Jiang,et al.  Indirect adaptive routing on large scale interconnection networks , 2009, ISCA '09.

[11]  Philip Heidelberger,et al.  The IBM Blue Gene/Q interconnection network and message unit , 2011, 2011 International Conference for High Performance Computing, Networking, Storage and Analysis (SC).

[12]  Torsten Hoefler,et al.  Enabling highly-scalable remote memory access programming with MPI-3 One Sided , 2014, Sci. Program..

[13]  Torsten Hoefler,et al.  Active Access: A Mechanism for High-Performance Distributed Data-Centric Computations , 2015, ICS.

[14]  William J. Dally,et al.  Flattened Butterfly Topology for On-Chip Networks , 2007, 40th Annual IEEE/ACM International Symposium on Microarchitecture (MICRO 2007).

[15]  William J. Dally,et al.  Principles and Practices of Interconnection Networks , 2004 .

[16]  John Shalf,et al.  Programming Abstractions for Data Locality , 2014 .

[17]  Torsten Hoefler,et al.  Deadlock-Free Oblivious Routing for Arbitrary Topologies , 2011, 2011 IEEE International Parallel & Distributed Processing Symposium.

[18]  José Duato,et al.  994 International Conference on Parallel Processing a Necessary and Sufficient Condition for Deadlock-free Adaptive Routing in Wormhole Networks , 2022 .

[19]  Vipin Kumar,et al.  A Fast and High Quality Multilevel Scheme for Partitioning Irregular Graphs , 1998, SIAM J. Sci. Comput..

[20]  Rudolf Lide,et al.  Finite fields , 1983 .

[21]  J. Sirán,et al.  Moore Graphs and Beyond: A survey of the Degree/Diameter Problem , 2013 .

[22]  Paul R. Hafner Geometric realisation of the graphs of McKay-Miller-Sirán , 2004, J. Comb. Theory, Ser. B.

[23]  Jana Siagiová,et al.  A Note on the McKay-Miller-Sira'n Graphs , 2001, J. Comb. Theory, Ser. B.

[24]  G. Farhi,et al.  Large Graphs with Given Degree and Diameter III , 1982 .

[25]  Ratko V. Tomic Optimal networks from error correcting codes , 2013, Architectures for Networking and Communications Systems.

[26]  Dennis Abts,et al.  Cray XT4 and Seastar 3-D Torus Interconnect , 2011, Encyclopedia of Parallel Computing.

[27]  Hong Liu,et al.  Energy proportional datacenter networks , 2010, ISCA.

[28]  Larry Kaplan,et al.  The Gemini System Interconnect , 2010, 2010 18th IEEE Symposium on High Performance Interconnects.

[29]  Sudhakar Yalamanchili,et al.  Interconnection Networks: An Engineering Approach , 2002 .

[30]  Charles Delorme,et al.  Grands Graphes de Degré et Diamètre Donnés , 1985, Eur. J. Comb..

[31]  Torsten Hoefler,et al.  The PERCS High-Performance Interconnect , 2010, 2010 18th IEEE Symposium on High Performance Interconnects.

[32]  Antonio Robles,et al.  A Survey and Evaluation of Topology-Agnostic Deterministic Routing Algorithms , 2012, IEEE Transactions on Parallel and Distributed Systems.

[33]  Xin Yuan,et al.  A new routing scheme for jellyfish and its performance with HPC workloads , 2013, 2013 SC - International Conference for High Performance Computing, Networking, Storage and Analysis (SC).

[34]  Michael M. Resch,et al.  Teraflops Sustained Performance With Real World Applications , 2008, Int. J. High Perform. Comput. Appl..

[35]  Torsten Hoefler,et al.  High-Performance Distributed RMA Locks , 2016, HPDC.

[36]  H. Niederreiter,et al.  Finite Fields: Encyclopedia of Mathematics and Its Applications. , 1997 .

[37]  William J. Dally,et al.  Deadlock-Free Message Routing in Multiprocessor Interconnection Networks , 1987, IEEE Transactions on Computers.

[38]  Charles E. Leiserson,et al.  Fat-trees: Universal networks for hardware-efficient supercomputing , 1985, IEEE Transactions on Computers.

[39]  Brendan D. McKay,et al.  A Note on Large Graphs of Diameter Two and Given Maximum Degree, , 1998, J. Comb. Theory, Ser. B.

[40]  Inder S. Gopal Prevention of Store-and-Forward Deadlock in Computer Networks , 1985, IEEE Trans. Commun..

[41]  Pedro López,et al.  Deterministic versus Adaptive Routing in Fat-Trees , 2007, 2007 IEEE International Parallel and Distributed Processing Symposium.

[42]  William J. Dally,et al.  Performance Analysis of k-Ary n-Cube Interconnection Networks , 1987, IEEE Trans. Computers.

[43]  Franz Franchetti,et al.  Mathematical foundations of the GraphBLAS , 2016, 2016 IEEE High Performance Extreme Computing Conference (HPEC).

[44]  Leslie G. Valiant,et al.  A Scheme for Fast Parallel Communication , 1982, SIAM J. Comput..

[45]  Béla Bollobás,et al.  Random Graphs: Notation , 2001 .

[46]  Abdou Youssef,et al.  Interconnection Networks for High-Performance Parallel Computers , 1994 .

[47]  William J. Dally,et al.  The BlackWidow High-Radix Clos Network , 2006, 33rd International Symposium on Computer Architecture (ISCA'06).

[48]  Amith R. Mamidala,et al.  Looking under the hood of the IBM Blue Gene/Q network , 2012, 2012 International Conference for High Performance Computing, Networking, Storage and Analysis.

[49]  Torsten Hoefler,et al.  Fault tolerance for remote memory access programming models , 2014, HPDC '14.

[50]  Alan M. Frieze,et al.  Random graphs , 2006, SODA '06.