The dynamic nature of Congestion inInfiniBand

The InfiniBand Congestion Control (CC) mechanism is able to reduce congestion and improve performance in many situations. In this paper we study the characteristics of congestion in InfiniBand by monitoring and analyzing the CC mechanism with a hardware analyzer. To the best of our knowledge, this is the first paper that presents experience with, and analysis of, the InfiniBand CC with such a tool. We found that there can be more than one “root of congestion”, as defined by the IBTA specification, existing at the same time in the congestion tree, and a “root of congestion” can be converted to a “victim of congestion” as its nature changes. We also observed that even with constant traffic flows, the “root of congestion” will shift from one place to another within the congestion tree, with corresponding consequence for packets from various traffic sources: traffic that might be negatively impacted by tree spreading and might be not contributing to the “root of congestion” before, will be treated as a congestion contributor and then be throttled by the CC mechanism.

[1]  Olav Lysne,et al.  Exploring the Scope of the InfiniBand Congestion Control Mechanism , 2012, 2012 IEEE 26th International Parallel and Distributed Processing Symposium.

[2]  Olav Lysne,et al.  On the Relation between Congestion Control, Switch Arbitration and Fairness , 2011, 2011 11th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing.

[3]  José Duato,et al.  FBICM: efficient congestion management for high-performance networks using distributed deterministic routing , 2008, HiPC'08.

[4]  José Duato,et al.  Efficient, Scalable Congestion Management for Interconnection Networks , 2006, IEEE Micro.

[5]  José Duato,et al.  An Efficient Strategy for Reducing Head-of-Line Blocking in Fat-Trees , 2010, Euro-Par.

[6]  Olav Lysne,et al.  First experiences with congestion control in InfiniBand hardware , 2010, 2010 IEEE International Symposium on Parallel & Distributed Processing (IPDPS).

[7]  José Duato,et al.  RECN-IQ: A Cost-Effective Input-Queued Switch Architecture with Congestion Management , 2007, 2007 International Conference on Parallel Processing (ICPP 2007).

[8]  José Duato,et al.  Cost-Effective Congestion Management for Interconnection Networks Using Distributed Deterministic Routing , 2010, 2010 IEEE 16th International Conference on Parallel and Distributed Systems.

[9]  Erich Strohmaier,et al.  TOP500 supercomputer , 2006, SC.

[10]  G. Pfister,et al.  Solving Hot Spot Contention Using InfiniBand Architecture Congestion Control , 2005 .

[11]  Gregory F. Pfister,et al.  “Hot spot” contention and combining in multistage interconnection networks , 1985, IEEE Transactions on Computers.

[12]  William J. Dally,et al.  Principles and Practices of Interconnection Networks , 2004 .

[13]  Samuel P. Morgan,et al.  Input Versus Output Queueing on a Space-Division Packet Switch , 1987, IEEE Trans. Commun..

[14]  Sven-Arne Reinemo,et al.  InfiniBand congestion control: modelling and validation , 2011, SimuTools.

[15]  José Duato,et al.  A new scalable and cost-effective congestion management strategy for lossless multistage interconnection networks , 2005, 11th International Symposium on High-Performance Computer Architecture.

[16]  José Duato,et al.  A new proposal to deal with congestion in InfiniBand-based fat-trees , 2014, J. Parallel Distributed Comput..

[17]  José Duato,et al.  Buffer Management Strategies to Reduce HoL Blocking , 2010, IEEE Transactions on Parallel and Distributed Systems.

[18]  José Duato,et al.  Dynamic Evolution of Congestion Trees: Analysis and Impact on Switch Architecture , 2005, HiPEAC.