Effects of Adaptive Wormhole Routing in Event Builder Networks

The data acquisition system of the CMS experiment at the Large Hadron Collider features a two-stage event builder, which combines data from about 500 sources into full events at an aggregate throughput of 100 GB/s. To meet the requirements, several architectures and interconnect technologies have been quantitatively evaluated. Myrinet will be used for the communication from the underground frontend devices to the surface event building system. Gigabit Ethernet is deployed in the surface event building system. Nearly full bi-section throughput can be obtained using a custom software driver for Myrinet based on barrel shifter traffic shaping. This paper discusses the use of Myrinet dual-port network interface cards supporting channel bonding to achieve virtual 5 GBit/s links with adaptive routing to alleviate the throughput limitations associated with wormhole routing. Adaptive routing is not expected to be suitable for high-throughput event builder applications in high-energy physics. To corroborate this claim, results from the CMS event builder preseries installation at CERN are presented and the problems of wormhole routing networks are discussed.

[1]  José Duato,et al.  Destination-based HoL blocking elimination , 2006, 12th International Conference on Parallel and Distributed Systems - (ICPADS'06).

[2]  Luciano Orsini,et al.  Software Architecture for Processing Clusters Based on I2O , 2004, Cluster Computing.

[3]  S. Cittolin,et al.  The CMS Event Builder , 2003 .

[4]  Laxmikant V. Kalé,et al.  Performance prediction using simulation of large-scale interconnection networks in POSE , 2005, Workshop on Principles of Advanced and Distributed Simulation (PADS'05).

[5]  Anoop Gupta,et al.  SPLASH: Stanford parallel applications for shared-memory , 1992, CARN.

[6]  Alessandro Genco,et al.  Peer-To-Peer Architectures in Distributed Data Management Systems for Large Hadron Collider Experiments , 2005 .

[7]  P. Geoffray.  Myrinet express (MX): Is your interconnect smart ? , 2004, Proceedings. Seventh International Conference on High Performance Computing and Grid in Asia Pacific Region, 2004..

[8]  Takaichi Yoshida,et al.  OORF: an object-oriented route finder , 2000, SAC '00.

[9]  A. W. Booth,et al.  Effects of various event building techniques on data acquisition system architectures , 1990 .

[10]  L.Berti,et al.  The CMS Event Builder Demonstrator and Results with Ethernet and Myrinet Switch Technologies , 2001 .

[11]  Henry M. Levy,et al.  Limits to low-latency communication on high-speed networks , 1993, TOCS.

[12]  Rajkumar Kettimuthu,et al.  A Performance Study of Parallel FFT in Clos and Mesh Networks , 2005, PDPTA.

[13]  Peter A. Franaszek,et al.  A DC-Balanced, Partitioned-Block, 8B/10B Transmission Code , 1983, IBM J. Res. Dev..

[14]  Charles L. Seitz,et al.  Myrinet: A Gigabit-per-Second Local Area Network , 1995, IEEE Micro.

[15]  Alain Ninane,et al.  Evaluation of Myrinet for the Event Builder of the CMS experiment , 2000 .

[16]  Luciano Orsini,et al.  Towards a homogeneous architecture for high-energy physics data acquisition systems , 2003 .

[17]  Fang Chen,et al.  A cluster-based active router architecture supporting video/audio stream transcoding service , 2003, Proceedings International Parallel and Distributed Processing Symposium.

[18]  J. Gutleber,et al.  The CMS event builder demonstrator and results with Myrinet , 2001 .

[19]  D. Samyn,et al.  The CMS event builder demonstrator based on Myrinet , 1999 .

[20]  Jack J. Dongarra,et al.  The LINPACK Benchmark: An Explanation , 1988, ICS.

[21]  Anoop Gupta,et al.  The SPLASH-2 programs: characterization and methodological considerations , 1995, ISCA.

[22]  Mark J. Karol,et al.  Queueing in high-performance packet switching , 1988, IEEE J. Sel. Areas Commun..

[23]  Dietmar Tutsch,et al.  MINSimulate - A MULTISTAGE INTERCONNECTION NETWORK SIMULATOR , 2003 .

[24]  Jeffrey S. Vetter,et al.  Performance Evaluation of High-Speed Interconnects Using Dense Communication Patterns , 2005, ICPP Workshops.