Exploring Design Challenges in Getting Solid State Drives Closer to CPU

The PCI Express Solid State Drives (PCIe SSDs) blur the difference between block and memory access semantic devices. Since these SSDs leverage PCIe bus as their storage interface, their interfaces are different from conventional memory system interconnects, as well as from standard storage interfaces. This leads to a new SSD architecture and storage software stack design. Unfortunately, there are not ample studies focusing on the system-level characteristics and the power consumption behaviors behind these emerging PCIe SSD platforms. In this paper, we quantitatively analyze the challenges faced by PCIe SSDs in getting flash memory closer to CPU, and study two representative PCIe SSD architectures (from-scratch SSD and bridge-based SSD) using state-of-the-art real SSDs with our in-house resource analyzer and dynamic evaluation platform. Our experimental analysis reveals that 1) while the from-scratch SSD approach offers remarkable performance improvements, it requires enormous host-side memory and computational resources; 2) the performance of the from-scratch SSD significantly degrades in a multi-core system; 3) bridge-based SSD architecture has a potential to improve performance by co-optimizing their flash software and controllers; 4) PCIe SSDs' latencies significantly degrade with their storage-level queueing mechanism; 5) tested PCIe SSDs consume 200 $\sim$ 500 percent more dynamic power than a conventional SSD; and 6) the power consumption increases by 33 percent, on average, when PCIe SSDs suffer from the performance drops caused by garbage collections. While our analytical study using the commercially available PCIe SSD products can include technical conjecture and inference, the findings and the empirical evidences present many design challenges that the future architecture and software stack probably need to address.