Memory Management Strategies for Data Serving with RDMA

Using remote direct memory access (RDMA) to ship data is becoming a very popular technique in network architectures. As these networks are adopted by the broader computing market, new challenges arise in transitioning existing code to use RDMA APIs. One particular class of applications that map poorly to RDMA are those that act as servers of file data. In order to access file data and send it over the network, an application must copy it to user-space buffers, and the operating system must register those buffers with the network adapter. Ordinary sockets-based networks can achieve higher performance by using the "sendfile" mechanism to avoid copying file data into user-space buffers. In this work we revisit time-honored approaches to sending file data, but adapted to RDMA networks. In particular, both pipelining and sendfile can be used, albeit with modifications to handle memory registration issues. However, memory registration is not well- integrated in current operating systems, leading to difficulties in adapting the sendfile mechanism. These two techniques make it feasible to create RDMA-based applications that serve file data and still maintain a high level of performance.