User Extensible Heap Manager for Heterogeneous Memory Platforms and Mixed Memory Policies

Memory management software requires additional sophistication for the array of new hardware technologies coming to market: on package addressable memory, stacked DRAM, nonvolatile high capacity DIMMs, and low-latency on-package fabric. As a complement to these hardware improvements there are many policy features that can be applied to virtual memory within the framework of the Linux system calls mmap(2), mbind(2), madvise(2), mprotect(2), and mlock(2). These policy features can support a wide range of future hardware capabilities including bandwidth control, latency control, inter-process sharing, inter-node sharing, accelerator sharing, persistence, checkpointing, and encryption. The combinatorial range implied by a platform with heterogeneous memory hardware, and many options for operating system policies applied to that hardware is enormous, so it is intractable to have a separate custom allocator addressing each of them. Each layer of the application software stack may have a variety of different requirements for memory properties. Some of those properties will be shared between clients, and some will be unique to the client. We propose software that will enable fine-grained client control over memory properties through our User Extensible Heap Manager, which efficiently reuses memory modified by expensive system calls and remains effective in a highly threaded environment.