A performance analysis of 4X InfiniBand data transfer operations

The performance of 4X InfiniBand send/receive and RDMA operations is studied by running tests to measure latency, data rate, number of operations per second, and CPU load. The measurements performed are for application-to-application data transfers using user-level InfiniBand (IB) verbs. It is shown that IB is capable of low latencies (10 μs for small messages) and very high data rates at low CPU loads (over 6 Gbs with 64 KB messages at under 20% CPU load). A very large number of operations per second (over 400,000) is obtained for small messages. Some comparisons are made with the performance of TCP/IP on Gigabit Ethernet. In addition, the paper studies the impact of varying the number of outstanding requests on the obtained throughput, and shows when the peak throughput can be obtained for messages of varying sizes. Finally, an approach for handling completions in user space without a busy wait and without the use of signals is introduced and CPU load results based on this approach are presented.