Title :
Designing high-performance and resilient message passing on InfiniBand
Author :
Koop, Matthew J. ; Shamis, Pavel ; Rabinovitz, Ishai ; Panda, Dhabaleswar K.
Author_Institution :
High Performance Technol., Inc. (HPTi), OH, USA
Abstract :
Clusters featuring the InfiniBand interconnect are continuing to scale. As an example, the ¿Ranger¿ system at the Texas Advanced Computing Center (TACC) includes over 60,000 cores with nearly 4,000 InfiniBand ports. The latest Top500 list shows 30% of systems and over 50% of the top 100 are now using InfiniBand as the compute node interconnect. As these systems continue to scale, the Mean-Time-Between-Failure (MTBF) is reducing and additional resiliency must be provided to the important components of HPC systems, including the MPI library. In this paper we present a design that leverages the reliability semantics of InfiniBand, but provides a higher-level of resiliency. We are able to avoid aborting jobs in the case of network failures as well as failures on the endpoints in the InfiniBand Host Channel Adapters (HCA). We propose reliability designs for rendezvous designs using both Remote DMA (RDMA) read and write operations. We implement a prototype of our design and show that performance is near-identical to that of a non-resilient design. This shows that we can have both the performance and the network reliability needed for large-scale systems.
Keywords :
message passing; InfiniBand host channel adapter; MPI library; Ranger system; high-performance message passing; mean-time-between-failure; reliability semantics; remote DMA; resilient message passing; Bandwidth; Computer science; Delay; Design engineering; Hardware; Large-scale systems; Libraries; Message passing; Prototypes; Quadratic programming;
Conference_Titel :
Parallel & Distributed Processing, Workshops and Phd Forum (IPDPSW), 2010 IEEE International Symposium on
Conference_Location :
Atlanta, GA
Print_ISBN :
978-1-4244-6533-0
DOI :
10.1109/IPDPSW.2010.5470856