vNUMA-mgr: Managing VM memory on NUMA platforms

Continuing improvements in the scale of many-core platforms are accompanied by increased asymmetry in their memory architectures. Such NUMA architectures, however, require systems software that understands this asymmetry to attain high levels of performance, leading to significant work in optimizing operating systems like Linux and Windows to increase locality of access to memory nodes and to consider differences in access latencies when accessing remote nodes. When running on today´s virtualized hardware platforms, however, virtual machines (VMs) remain unaware of underlying memory architecture, leading to non-compliance with their operating system´s (OS) policies for managing NUMA memory, and thereby, incurring undesirable performance overheads. This paper describes the vNUMA-mgr approach of dealing with the NUMA memory characteristics of future virtualized multicore architectures, which (1) properly manages VM memory at the hyper-visor (VMM) level through a set of VMM-level strategies, coupled with (2) ´enlightening´ guest VM OSes, if possible, to aid their memory management, and also (3) provides mechanisms to maintain the distribution of VM memory (across physical nodes), even when memory is over-provisioned. The vNUMA-mgr implementation in the Xen VMM is evaluated with respect to each of its memory allocation strategies. The evaluation, which uses a set of memory-intensive benchmarks representative of many High-Performance Computing (HPC) applications, shows 30-50% performance improvement on our platform.