FRAMP: a fast remote direct memory access and message passing network

Interconnect network, host interface, and communication protocol determine the overall performance of message passing on cluster systems. Most previous work only focuses on these aspects separately. This paper describes the design of FRAMP (virtual memory based fast remote direct memory access and message passing network), including the crossbar-based switch, the custom host network interface and the user-level protocol for cluster communication. By deliberately balancing all three aspects and their interaction, FRAMP achieves 5.6 μs one-way latency and 2.0 μs ping-ping latency and 125 MB/s asymptotic bandwidth with message passing. Remote memory read for a 4096-byte data only takes about 39 μs. These are close to the hardware limit of our experimental environment, which is based on 33 MHz 32-bit PCI bus. All figures about system performance are measured at the communication library API level, which can be directly and fully utilized by user applications.