Hiding the Microsecond-Scale Latency of Storage-Class Memories with Duplexity

We are entering the “killer microsecond” era in data center applications [1]. Due to advances in processor, memory, storage, and networking technologies, events that stall execution increasingly fall in a microsecond-scale latency range [2, 3]. Storage-class memories, such as 3D Xpoint, are examples of such events that stall execution for single-digit microseconds. Whereas contemporary computing systems are effectively equipped with mechanisms to hide nanosecondand millisecond-scale stalls, they lack efficient support for microsecond-scale stalls [1]. Nanosecond-scale stalls are effectively hidden by microarchitectural mechanisms, such as Out-of-Order (OoO) execution and deep memory hierarchies, but these mechanisms are insufficient to hide microsecond-scale stalls. Conversely, operating systems use context switching to hide millisecond-scale latencies, such as when accessing disk. However, context switch overheads (5-20μs [4]) are within the same latency orders as microsecond-scale stalls, so they are not a plausible latency-hiding technique for the microsecond regime. Simultaneous multithreading (SMT) has been proposed to colocate latency-critical and batch threads on the same core so that the batch threads fill the utilization holes caused by brief I/O stalls or inter-request idle time [5, 6]. Already today, scale-out workloads deployed in data centers exhibit low CPU utilization due to lack of memory level parallelism and front-end inefficiencies, calling for more SMT threads even in the absence of μs-scale stalls [7]. As batch workloads also adopt mechanisms like storage-class memory or rack-scale disaggregation, these workloads, too, will incur such stalls. As a consequence, even more threads must be added to ensure that, at any time, there are enough unstalled threads to fill a core’s available execution bandwidth—the two threads offered by Intel’s hyper-threading are not nearly enough. Unfortunately, scaling SMT microarchitecture to support many more threads is prohibitive, due to high logic complexity, wire delay, limited register file (RF) capacity, and cache pressure/thrashing among threads. Moreover, as previous studies have shown [8], some SMT thread co-locations can have catastrophic impact on the tails of latency-critical threads, especially at high loads, due to contention for shared resources [9]. To avoid compromising the tail latency of critical threads due to SMT interference, we instead design Duplexity [10], a server architecture that seeks directly to address the killer-microsecond challenge—to fill in the μs-scale “holes” in threads’ execution schedules, which arise due to idleness and stalls, with useful execution, without impacting the tail latency of latency-critical threads. Duplexity is the first server architecture that aims to improve server utilization in the presence of μs-scale stalls, without sacrificing QoS and tail latency of micro-services. Our evaluation, using Gem5 [11] and BigHouse [12] simulation frameworks, demonstrates that Duplexity can improve core utilization by 4.8× and 1.9×, and iso-throughput 99th-percentile tail latency by 1.8× and 2.7×, on average, over a baseline OoO and an SMT-based server architecture, respectively.