Simulated Performance Evaluation of a 6D Mesh/iBT Interconnect

A new 6D mesh/torus (Tofu) interconnect is implemented in the K Computer for achieving a performance of more than 10 petaflops. As expected, among its many interesting properties, such interconnect has large variations of latencies for communications within a node-group (intra-node-group) or between them (inter-node-group). Inspired by their works while recognizing the superior features of the recently published interlaced bypass torus (iBT) network that can help reduce such variations, we propose a new 6D mesh/iBT interconnect by replacing the 3D torus of Tofu by a 3D iBT. Our new interconnect suppresses the latencies for inter-node-group communications to dramatically improve the overall performance for the entire network. Adapting an industrial standard network simulator NS-3, we study different communication patterns of the added iBT networks under different bypass schemes. We find that the simulated average latencies of our network are 40% less than those of the Tofu interconnect for all tested patterns.

[1]  Toshiyuki Shimizu,et al.  Tofu: A 6D Mesh/Torus Interconnect for Exascale Computers , 2009, Computer.

[2]  Laxmikant V. Kalé,et al.  Avoiding hot-spots on two-level direct networks , 2011, 2011 International Conference for High Performance Computing, Networking, Storage and Analysis (SC).

[3]  Larry Kaplan,et al.  The Gemini System Interconnect , 2010, 2010 18th IEEE Symposium on High Performance Interconnects.

[4]  George F. Riley,et al.  The ns-3 Network Simulator , 2010, Modeling and Tools for Network Simulation.

[5]  Sudhakar Yalamanchili,et al.  Interconnection Networks: An Engineering Approach , 2002 .

[6]  Xuejun Yang,et al.  Implementation and Evaluation of Network Interface and Message Passing Services for TianHe-1A Supercomputer , 2011, 2011 IEEE 19th Annual Symposium on High Performance Interconnects.

[7]  John Shalf,et al.  Exascale Computing Technology Challenges , 2010, VECPAR.

[8]  Philip Heidelberger,et al.  Blue Gene/L torus interconnection network , 2005, IBM J. Res. Dev..

[9]  Tomohiro Inoue,et al.  The Tofu Interconnect , 2011, 2011 IEEE 19th Annual Symposium on High Performance Interconnects.

[10]  Fumiyoshi Shoji,et al.  The K computer: Japanese next-generation supercomputer development project , 2011, IEEE/ACM International Symposium on Low Power Electronics and Design.

[11]  Ali Pinar,et al.  A Simulator for Large-Scale Parallel Computer Architectures , 2010, Int. J. Distributed Syst. Technol..

[12]  Philip Heidelberger,et al.  The IBM Blue Gene/Q interconnection network and message unit , 2011, 2011 International Conference for High Performance Computing, Networking, Storage and Analysis (SC).

[13]  Peng Zhang,et al.  Interlacing Bypass Rings to Torus Networks for More Efficient Networks , 2011, IEEE Transactions on Parallel and Distributed Systems.

[14]  Torsten Hoefler,et al.  The PERCS High-Performance Interconnect , 2010, 2010 18th IEEE Symposium on High Performance Interconnects.

[15]  William J. Dally,et al.  Principles and Practices of Interconnection Networks , 2004 .