Energy-Proportional Data Center Network Architecture Through OS, Switch and Laser Co-design

Optical interconnects are already the dominant technology in large-scale data center networks. However, the high optical loss of many optical components coupled with the low efficiency of laser sources result in high aggregate power requirements for the thousands of optical transceivers used by these networks. As optical interconnects stay always on even as traffic demands ebb and flow, most of this power is wasted. We present LC DC , a data center network system architecture in which the operating system, the switch, and the optical components are co-designed to achieve energy proportionality. LC DC capitalizes on the path divergence of data center networks to turn on and off redundant paths according to traffic demand, while maintaining full connectivity. Turning off redundant paths allows the optical transceivers and their electronic drivers to power down and save energy. Maintaining full connectivity hides the laser turn-on delay. At the node layer, intercepting send requests within the OS allows for the NIC’s laser turn-on delay to be fully overlapped with TCP/IP packet processing, and thus egress links can remain powered off until needed with zero performance penalty. We demonstrate the feasibility of LC DC by i) implementing the necessary modifications in the Linux kernel and device drivers, ii) implementing a 10 Gbit/s FPGA switch, and iii) performing physical experiments with optical devices and circuit simulations. Our results on university data center traces and models of Facebook and Microsoft data center traffic show that LC DC saves on average 60% of the optical transceivers power (68% max) at the cost of 6% higher packet delay.

[1]  Randy H. Katz,et al.  Greening the Switch , 2008, HotPower.

[2]  Nikolaos Hardavellas,et al.  EcoLaser: An adaptive laser control for energy-efficient on-chip photonic interconnects , 2014, 2014 IEEE/ACM International Symposium on Low Power Electronics and Design (ISLPED).

[3]  Albert G. Greenberg,et al.  VL2: a scalable and flexible data center network , 2009, SIGCOMM '09.

[4]  Margaret Martonosi,et al.  Wattch: a framework for architectural-level power analysis and optimizations , 2000, Proceedings of 27th International Symposium on Computer Architecture (IEEE Cat. No.RS00201).

[5]  Daniel Hagimont,et al.  Welcome to zombieland: practical and energy-efficient memory disaggregation in a datacenter , 2018, EuroSys.

[6]  Amin Vahdat,et al.  Helios: a hybrid electrical/optical switch architecture for modular data centers , 2010, SIGCOMM '10.

[7]  David A. Maltz,et al.  Network traffic characteristics of data centers in the wild , 2010, IMC '10.

[8]  Xuan Zhang,et al.  Near-Memory Processing in Action: Accelerating Personalized Recommendation With AxDIMM , 2021, IEEE Micro.

[9]  Ram Huggahalli,et al.  Architectural Breakdown of End-to-End Latency in a TCP/IP Network , 2007, 19th International Symposium on Computer Architecture and High Performance Computing (SBAC-PAD'07).

[10]  Nikos Hardavellas,et al.  LaC: Integrating laser control in a photonic interconnect , 2014, 2014 IEEE Photonics Conference.

[11]  Hong Liu,et al.  Energy proportional datacenter networks , 2010, ISCA.

[12]  Ralph H. Johnson,et al.  Emerging VCSEL technologies at Finisar , 2010, OPTO.

[13]  Luiz André Barroso,et al.  The Datacenter as a Computer: An Introduction to the Design of Warehouse-Scale Machines , 2009, The Datacenter as a Computer: An Introduction to the Design of Warehouse-Scale Machines.

[14]  Amin Vahdat,et al.  Data Center Switch Architecture in the Age of Merchant Silicon , 2009, 2009 17th IEEE Symposium on High Performance Interconnects.

[15]  Ndubuisi G. Orji,et al.  Virtual Metrology White Paper - INTERNATIONAL ROADMAP FOR DEVICES AND SYSTEMS(IRDS) , 2018 .

[16]  Mahmut T. Kandemir,et al.  Evaluating STT-RAM as an energy-efficient main memory alternative , 2013, 2013 IEEE International Symposium on Performance Analysis of Systems and Software (ISPASS).

[17]  Albert G. Greenberg,et al.  The nature of data center traffic: measurements & analysis , 2009, IMC '09.

[18]  Nikos Hardavellas,et al.  Towards energy-efficient photonic interconnects , 2015, Photonics West - Optoelectronic Materials and Devices.

[19]  J. Thomas Pawlowski,et al.  Hybrid memory cube (HMC) , 2011, 2011 IEEE Hot Chips 23 Symposium (HCS).

[20]  Nick McKeown,et al.  Matching output queueing with a combined input output queued switch , 1999, IEEE INFOCOM '99. Conference on Computer Communications. Proceedings. Eighteenth Annual Joint Conference of the IEEE Computer and Communications Societies. The Future is Now (Cat. No.99CH36320).

[21]  Sujata Banerjee,et al.  ElasticTree: Saving Energy in Data Center Networks , 2010, NSDI.

[22]  William J. Dally,et al.  Principles and Practices of Interconnection Networks , 2004 .

[23]  Steven Swanson,et al.  Rethinking Flash in the Data Center , 2010, IEEE Micro.

[24]  Radu Sion,et al.  DIMMer: A case for turning off DIMMs in clouds , 2014, SoCC.

[25]  Kevin Skadron,et al.  Power-aware computing , 2003, Computer.

[26]  Eric S. Chung,et al.  A reconfigurable fabric for accelerating large-scale datacenter services , 2014, 2014 ACM/IEEE 41st International Symposium on Computer Architecture (ISCA).

[27]  Lizhong Chen,et al.  MP3: Minimizing performance penalty for power-gating of Clos network-on-chip , 2014, 2014 IEEE 20th International Symposium on High Performance Computer Architecture (HPCA).

[28]  Nikolaos Hardavellas,et al.  SLaC: Stage laser control for a flattened butterfly network , 2016, 2016 IEEE International Symposium on High Performance Computer Architecture (HPCA).

[29]  Hong Liu,et al.  Jupiter Rising: A Decade of Clos Topologies and Centralized Control in Google's Datacenter Network , 2015, Comput. Commun. Rev..

[30]  Georgios Zervas,et al.  Synchronous subnanosecond clock and data recovery for optically switched data centres using clock phase caching , 2020, Nature Electronics.

[31]  Mounir Meghelli,et al.  A 25 Gb/s burst-mode receiver for low latency photonic switch networks , 2015, 2015 Optical Fiber Communications Conference and Exhibition (OFC).

[32]  Jaehyuk Huh,et al.  Charge-Aware DRAM Refresh Reduction with Value Transformation , 2020, 2020 IEEE International Symposium on High Performance Computer Architecture (HPCA).

[33]  Hitesh Ballani,et al.  Sirius: A Flat Datacenter Network with Nanosecond Optical Switching , 2020, SIGCOMM.

[34]  Polina Bayvel,et al.  Sub-Nanosecond Clock and Data Recovery in an Optically-Switched Data Centre Network , 2018, 2018 European Conference on Optical Communication (ECOC).

[35]  Wolf-Dietrich Weber,et al.  Power provisioning for a warehouse-sized computer , 2007, ISCA '07.

[36]  Alex C. Snoeren,et al.  Inside the Social Network's (Datacenter) Network , 2015, Comput. Commun. Rev..

[37]  Shekhar Y. Borkar Exascale Computing - A Fact or a Fiction? , 2013, IPDPS.

[38]  R Baets,et al.  Flip-chip assembly of VCSELs to silicon grating couplers via laser fabricated SU8 prisms. , 2015, Optics express.

[39]  Luiz André Barroso,et al.  The Case for Energy-Proportional Computing , 2007, Computer.