Evaluation of a Disaggregated Rack System

NexTCA is an industrial-strength disaggregated rack system that is designed to provide a flexible computing infrastructure for heterogeneous workloads with wildly different resource demands. The key innovation in NexTCA is a PCI Express (PCIe) network that connects a rack of servers in such a way that the DRAMs, network interface cards (NIC), and SAS or NVMe solid state disks (SSD) associated with one server are directly accessible to other servers in the same rack. Leveraging this novel flexibility, NexTCA organizes the DRAMs, NICs and SSDs anywhere in a NexTCA rack into a global DRAM, NIC and SSD pool, respectively, and dynamically allocates hardware resources to individual servers that may or may not host the allocated resources. That is, NexTCA enables the notion of software-defined server, where the set of hardware resources bound to a server is configurable via software at run time, rather than fixed at manufacturing time. In addition, NexTCA uses the same PCIe network to support high-performance intra-rack inter-server communications. This paper starts with a brief description of the hardware architecture and systems software stack of NexTCA, and proceeds with a detailed evaluation of the first NexTCA prototype, which is jointly developed by ADLINK Technology and Industrial Technology Research Institute.

[1]  Venkata Krishnan,et al.  PCI express and advanced switching: evolutionary path to building next generation interconnects , 2003, 11th Symposium on High Performance Interconnects, 2003. Proceedings..

[2]  Venkata Krishnan.  Towards an integrated IO and clustering solution using PCI express , 2007, 2007 IEEE International Conference on Cluster Computing.

[3]  Mitsuhisa Sato,et al.  PEARL: Power-Aware, Dependable, and High-Performance Communication Link Using PCI Express , 2010, 2010 IEEE/ACM Int'l Conference on Green Computing and Communications & Int'l Conference on Cyber, Physical and Social Computing.

[4]  Jason Taylor,et al.  Facebook's data center infrastructure: Open compute, disaggregated rack, and beyond , 2015, 2015 Optical Fiber Communications Conference and Exhibition (OFC).

[5]  John Byrne,et al.  Power-efficient networking for balanced system designs: early experiences with PCIe , 2011, HotPower '11.

[6]  Tzi-cker Chiueh,et al.  Secure I/O device sharing among virtual machines on multiple hosts , 2013, ISCA.

[7]  Richard B. Gillett.  Memory Channel Network for PCI , 1996, IEEE Micro.

[8]  Venkata Krishnan.  Evaluation of an Integrated PCI Express IO Expansion and Clustering Fabric , 2008, 2008 16th IEEE Symposium on High Performance Interconnects.

[9]  Tzi-cker Chiueh,et al.  Marlin: A memory-based rack area network , 2014, 2014 ACM/IEEE Symposium on Architectures for Networking and Communications Systems (ANCS).

[10]  Kwok Kong,et al.  Application Note AN-571 PCI Express® System Interconnect Software Architecture for x86-based Systems , 2007 .

[11]  Richard Kaufmann,et al.  Using the Memory Channel Network , 1997, IEEE Micro.