Minimal Rewiring: Efficient Live Expansion for Clos Data Center Networks

Clos topologies have been widely adopted for large-scale data center networks (DCNs), but it has been difficult to support incremental expansions for Clos DCNs. Some prior work has claimed that the structure of Clos topologies hinders incremental expansion. We demonstrate that it is indeed possible to design expandable Clos DCNs, and to expand them while they are carrying live traffic, without incurring packet loss. We use a layer of patch panels between blocks of switches in a Clos DCN, which makes physical rewiring feasible, and we describe how to use integer linear programming (ILP) to minimize the number of patch-panel connections that must be changed, which makes expansions faster and cheaper. We also describe a block-aggregation technique that makes our ILP approach scalable. We tested our “minimal-rewiring” solver on two kinds of fine-grained expansions using 2250 synthetic DCN topologies, and found that the solver can handle 99% of these cases while changing under 25% of the connections. Compared to prior approaches, this solver (on average) reduces the average number of “stages” per expansion from 4 to 1.29, and reduces the number of wires changed by an order of magnitude or more – a significant improvement to our operational costs, and to our exposure (during expansions) to capacity-reducing faults.

[1]  Charles Clos,et al.  A study of non-blocking switching networks , 1953 .

[2]  Richard M. Karp,et al.  Theoretical Improvements in Algorithmic Efficiency for Network Flow Problems , 1972, Combinatorial Optimization.

[3]  L. Trotter,et al.  Integer Rounding and Polyhedral Decomposition for Totally Unimodular Systems , 1978 .

[4]  Andrew V. Goldberg,et al.  A new approach to the maximum flow problem , 1986, STOC '86.

[5]  Sanguthevar Rajasekaran,et al.  On the Convergence Time of Simulated Annealing , 1990 .

[6]  Mark Jerrum,et al.  Three-Dimensional Statistical Data Security Problems , 1994, SIAM J. Comput..

[7]  Jens Vygen,et al.  Efficient implementation of the Goldberg–Tarjan minimum-cost flow algorithm , 1998 .

[8]  Stephen P. Boyd,et al.  Branch and Bound Methods , 1987 .

[9]  Amin Vahdat,et al.  A scalable, commodity data center network architecture , 2008, SIGCOMM '08.

[10]  Lei Shi,et al.  Dcell: a scalable and fault-tolerant network structure for data centers , 2008, SIGCOMM '08.

[11]  Albert G. Greenberg,et al.  VL2: a scalable and flexible data center network , 2009, SIGCOMM '09.

[12]  Haitao Wu,et al.  FiConn: Using Backup Port for Server Interconnection in Data Centers , 2009, IEEE INFOCOM 2009.

[13]  Haitao Wu,et al.  BCube: a high performance, server-centric network architecture for modular data centers , 2009, SIGCOMM '09.

[14]  Amin Vahdat,et al.  Helios: a hybrid electrical/optical switch architecture for modular data centers , 2010, SIGCOMM '10.

[15]  Alejandro López-Ortiz,et al.  LEGUP: using heterogeneity to reduce the cost of data center network upgrades , 2010, CoNEXT.

[16]  Srinivasan Keshav,et al.  REWIRE: An Optimization-based Framework for Data Center Network Design , 2011 .

[17]  Ankit Singla,et al.  Jellyfish: Networking Data Centers Randomly , 2011, NSDI.

[18]  Amin Vahdat,et al.  Aspen trees: balancing data center fault tolerance, scalability and cost , 2013, CoNEXT.

[19]  Balázs Sonkoly,et al.  Free-scaling your data center , 2013, Comput. Networks.

[20]  Nathan Farrington,et al.  Facebook's data center network architecture , 2013, 2013 Optical Interconnects Conference.

[21]  A. Krishnamurthy,et al.  F10: A Fault-Tolerant Engineered Network , 2013, NSDI.

[22]  Yunhao Liu,et al.  Expandable and Cost-Effective Network Structures for Data Centers Using Dual-Port Servers , 2013, IEEE Transactions on Computers.

[23]  Thomas E. Anderson,et al.  F10: A Fault-Tolerant Engineered Network , 2013, NSDI.

[24]  Ye Yu,et al.  Space Shuffle: A Scalable, Flexible, and High-Bandwidth Data Center Network , 2014, ICNP.

[25]  Min Zhu,et al.  WCMP: weighted cost multipathing for improved fairness in data centers , 2014, EuroSys '14.

[26]  Minlan Yu,et al.  Condor: Better Topologies Through Declarative Design , 2015, Comput. Commun. Rev..

[27]  Michael Dinitz,et al.  Explicit Expanding Expanders , 2015, ESA.

[28]  Michael Dinitz,et al.  Xpander: Towards Optimal-Performance Datacenters , 2016, CoNEXT.

[29]  Hong Liu,et al.  Jupiter Rising: A Decade of Clos Topologies and Centralized Control in Google's Datacenter Network , 2015, Comput. Commun. Rev..

[30]  Yuanyuan Yang,et al.  BCCC: An Expandable Network for Data Centers , 2014, IEEE/ACM Transactions on Networking.

[31]  Ramón Beivide,et al.  Random Folded Clos Topologies for Datacenter Networks , 2017, 2017 IEEE International Symposium on High Performance Computer Architecture (HPCA).