Sirius: A Flat Datacenter Network with Nanosecond Optical Switching

The increasing gap between the growth of datacenter traffic and electrical switch capacity is expected to worsen due to the slowdown of Moore's law, motivating the need for a new switching technology for the post-Moore's law era that can meet the increasingly stringent requirements of hardware-driven cloud workloads. We propose Sirius, an optically-switched network for datacenters providing the abstraction of a single, high-radix switch that can connect thousands of nodes---racks or servers---in a datacenter while achieving nanosecond-granularity reconfiguration. At its core, Sirius uses a combination of tunable lasers and simple, passive gratings that route light based on its wavelength. Sirius' switching technology and topology is tightly codesigned with its routing and scheduling and with novel congestion-control and time-synchronization mechanisms to achieve a scalable yet flat network that can offer high bandwidth and very low end-to-end latency. Through a small-scale prototype using a custom tunable laser chip that can tune in less than 912 ps, we demonstrate 3.84 ns end-to-end reconfiguration atop 50 Gbps channels. Through large-scale simulations, we show that Sirius can approximate the performance of an ideal, electrically-switched non-blocking network with up to 74-77% lower power.

[1]  Srinivasan Keshav,et al.  Quartz , 2014, SIGCOMM.

[2]  Konstantina Papagiannaki,et al.  c-Through: part-time optics in data centers , 2010, SIGCOMM '10.

[3]  Alex C. Snoeren,et al.  P-FatTree: A multi-channel datacenter network topology , 2016, HotNets.

[4]  T. S. Eugene Ng,et al.  Enabling Topological Flexibility for Data Centers Using OmniSwitch , 2015, HotCloud.

[5]  Brian Corbett,et al.  A 10Gbps optical burst switching network incorporating ultra-fast (5ns) wavelength switched tunable laser sources , 2017, International Conference on Space Optics.

[6]  He Liu,et al.  Circuit Switching Under the Radar with REACToR , 2014, NSDI.

[7]  Ming C. Wu,et al.  Large-scale broadband digital silicon photonic switches with vertical adiabatic couplers , 2016 .

[8]  Ankit Singla,et al.  OSA: An Optical Switching Architecture for Data Center Networks With Unprecedented Flexibility , 2012, IEEE/ACM Transactions on Networking.

[9]  Hitesh Ballani,et al.  Bridging the Last Mile for Optical Switching in Data Centers , 2018, 2018 Optical Fiber Communications Conference and Exposition (OFC).

[10]  Alex C. Snoeren,et al.  Inside the Social Network's (Datacenter) Network , 2015, Comput. Commun. Rev..

[11]  H. Jonathan Chao,et al.  On the performance of a dual round-robin switch , 2001, Proceedings IEEE INFOCOM 2001. Conference on Computer Communications. Twentieth Annual Joint Conference of the IEEE Computer and Communications Society (Cat. No.01CH37213).

[12]  Georgios Zervas,et al.  PULSE: Optical Circuit Switched Data Center Architecture Operating at Nanosecond Timescales , 2020, Journal of Lightwave Technology.

[13]  Alex C. Snoeren,et al.  RotorNet: A Scalable, Low-complexity, Optical Datacenter Network , 2017, SIGCOMM.

[14]  Chunming Qiao,et al.  Enabling Wide-Spread Communications on Optical Fabric with MegaSwitch , 2017, NSDI.

[15]  Arvind Krishnamurthy,et al.  High-resolution measurement of data center microbursts , 2017, Internet Measurement Conference.

[16]  Ramana Rao Kompella,et al.  On the impact of packet spraying in data center networks , 2013, 2013 Proceedings IEEE INFOCOM.

[17]  L.A. Coldren,et al.  Tunable semiconductor lasers: a tutorial , 2004, Journal of Lightwave Technology.

[18]  Rajdeep Das,et al.  Expanding across time to deliver bandwidth efficiency and low latency , 2019, NSDI.

[19]  Albert G. Greenberg,et al.  Data center TCP (DCTCP) , 2010, SIGCOMM '10.

[20]  Gal Shahaf,et al.  Beyond fat-trees without antennae, mirrors, and disco-balls , 2017, SIGCOMM.

[21]  Hitesh Ballani,et al.  Hybrid Wavelength Switched-TDMA High Port Count All-Optical Data Centre Switch , 2017, Journal of Lightwave Technology.

[22]  Leslie G. Valiant,et al.  Universal schemes for parallel communication , 1981, STOC '81.

[23]  Nikhil R. Devanur,et al.  ProjecToR: Agile Reconfigurable Data Center Interconnect , 2016, SIGCOMM.

[24]  Qixiang Cheng,et al.  Highly-scalable, low-crosstalk architecture for ring-based optical space switch fabrics , 2017, 2017 IEEE Optical Interconnects Conference (OI).

[25]  Amin Vahdat,et al.  Switching the optical divide: fundamental challenges for hybrid electrical/optical datacenter networks , 2011, SoCC.

[26]  Alexander V. Rylyakov,et al.  Monolithic Silicon Integration of Scaled Photonic Switch Fabrics, CMOS Logic, and Device Driver Circuits , 2014, Journal of Lightwave Technology.

[27]  Albert G. Greenberg,et al.  VL2: a scalable and flexible data center network , 2009, SIGCOMM '09.

[28]  M. Duser,et al.  Analysis of a dynamically wavelength-routed optical burst switched network architecture , 2002 .

[29]  P. Kozodoy,et al.  Performance and reliability of widely tunable laser diodes , 2003, 53rd Electronic Components and Technology Conference, 2003. Proceedings..

[30]  Paramvir Bahl,et al.  Augmenting data center networks with multi-gigabit wireless links , 2011, SIGCOMM.

[31]  Hong Liu,et al.  Jupiter Rising: A Decade of Clos Topologies and Centralized Control in Google's Datacenter Network , 2015, Comput. Commun. Rev..

[32]  Samuel K. Moore,et al.  Another step toward the end of Moore's law: Samsung and TSMC move to 5-nanometer manufacturing - [News] , 2019, IEEE Spectrum.

[33]  M. Glick,et al.  Optical switching for next generation data centers , 2009, 2009 International Conference on Photonics in Switching.

[34]  Binhao Wang,et al.  50 Gb/s PAM4 Low-Voltage Si-Ge Avalanche Photodiode , 2019, 2019 Conference on Lasers and Electro-Optics (CLEO).

[35]  Hua Chen,et al.  Pingmesh: A Large-Scale System for Data Center Network Latency Measurement and Analysis , 2015, SIGCOMM.

[36]  Himanshu Shah,et al.  FireFly , 2014, SIGCOMM.

[37]  ChangCheng-Shang,et al.  Load balanced Birkhoff-von Neumann switches, part I , 2002 .

[38]  Francesco Testa,et al.  Optical Switching in Next Generation Data Centers , 2018 .

[39]  Amin Vahdat,et al.  Less Is More: Trading a Little Bandwidth for Ultra-Low Latency in the Data Center , 2012, NSDI.

[40]  Amin Vahdat,et al.  Exploiting a Natural Network Effect for Scalable, Fine-grained Clock Synchronization , 2018, NSDI.

[41]  Maciej Lipinski,et al.  White rabbit clock characteristics , 2016, 2016 IEEE International Symposium on Precision Clock Synchronization for Measurement, Control, and Communication (ISPCS).

[42]  F. Karinou,et al.  System Demonstration of Nanosecond Wavelength Switching with Burst-mode PAM4 Transceiver , 2019, 45th European Conference on Optical Communication (ECOC 2019).

[43]  Hung-Hsiang Jonathan Chao,et al.  Centralized contention resolution schemes for a large-capacity optical ATM switch , 1998, 1998 IEEE ATM Workshop Proceedings. 'Meeting the Challenges of Deploying the Global Broadband Network Infrastructure' (Cat. No.98EX164).

[44]  Ashok V. Krishnamoorthy,et al.  Computer Systems Based on Silicon Photonic Interconnects A proposed supercomputer-on-a-chip with optical interconnections between processing elements will require development of new lower-energy optical components and new circuit architectures that match electrical datapaths to complementary optical , 2009 .

[45]  Amin Vahdat,et al.  Helios: a hybrid electrical/optical switch architecture for modular data centers , 2010, SIGCOMM '10.

[46]  Polina Bayvel,et al.  Sub-Nanosecond Clock and Data Recovery in an Optically-Switched Data Centre Network , 2018, 2018 European Conference on Optical Communication (ECOC).

[47]  Benn C. Thomsen,et al.  Sub-Nanosecond Optical Switching using Chip-Based Soliton Microcombs , 2020, 2020 Optical Fiber Communications Conference and Exhibition (OFC).

[48]  Nicola Calabretta,et al.  Monolithically integrated WDM cross-connect switch for high-performance optical data center networks , 2017, 2017 Optical Fiber Communications Conference and Exhibition (OFC).

[49]  Miles H. Anderson,et al.  Microresonator-based solitons for massively parallel coherent optical communications , 2016, Nature.

[50]  S. J. B. Yoo,et al.  Ultra-Compact Silicon Photonic 512 × 512 25 GHz Arrayed Waveguide Grating Router , 2014, IEEE Journal of Selected Topics in Quantum Electronics.

[51]  Srinivasan Seshan,et al.  Scheduling techniques for hybrid circuit/packet networks , 2015, CoNEXT.

[52]  Amin Vahdat,et al.  Integrating microsecond circuit switching into the data center , 2013, SIGCOMM.

[53]  Georgios Zervas,et al.  Synchronous subnanosecond clock and data recovery for optically switched data centres using clock phase caching , 2020, Nature Electronics.

[54]  Georgios Zervas,et al.  SWIFT: Scalable Ultra-Wideband Sub-Nanosecond Wavelength Switching for Data Centre Networks , 2020, ArXiv.

[55]  Ben Y. Zhao,et al.  Mirror mirror on the ceiling: flexible wireless links for data centers , 2012, CCRV.

[56]  R. Proietti,et al.  Experimental demonstration of a 64-port wavelength routing thin-CLOS system for data center switching architectures , 2018, IEEE/OSA Journal of Optical Communications and Networking.

[57]  V. A. Shchukin,et al.  4×40 Gb/s 2 pJ/bit Optical RX with 8ns Power-on and CDR-Lock Time in 14nm CMOS , 2018, 2018 Optical Fiber Communications Conference and Exposition (OFC).

[58]  Hakim Weatherspoon,et al.  Shoal: A Network Architecture for Disaggregated Racks , 2019, NSDI.

[59]  A Wonfor,et al.  Demonstration of the feasibility of large-port-count optical switching using a hybrid Mach-Zehnder interferometer-semiconductor optical amplifier switch module in a recirculating loop. , 2014, Optics letters.

[60]  Hakim Weatherspoon,et al.  Globally Synchronized Time via Datacenter Networks , 2016, SIGCOMM.

[61]  Yasuyoshi Uchida,et al.  LCOS-based flexible grid 1×40 wavelength selective switch using planar lightwave circuit as spot size converter , 2015, 2015 Optical Fiber Communications Conference and Exhibition (OFC).

[62]  Qixiang Cheng,et al.  Photonic switching in high performance datacenters [Invited]. , 2018, Optics express.

[63]  Xian Xiao,et al.  Experimental Demonstration of Flex-LIONS for Reconfigurable All-to-All Optical Interconnects , 2018, 2018 European Conference on Optical Communication (ECOC).

[64]  Hua Chen,et al.  Pingmesh: A Large-Scale System for Data Center Network Latency Measurement and Analysis , 2015, SIGCOMM.

[65]  Nick McKeown,et al.  pFabric: minimal near-optimal datacenter transport , 2013, SIGCOMM.