A software architecture for global address space communication on clusters: put/get on fast messages

Global address space parallel programming models can be an effective alternative to send/receive style communication, simplifying programming or code generation and increasing performance for certain application types. Traditionally, global address space mechanisms have been implemented in hardware in order to provide the necessary communication performance and responsiveness. However new high performance cluster messaging systems now allow global address space mechanisms to be realized efficiently in software. We describe a high performance one sided communication model that is implemented as a software layer on top of the Illinois Fast Messages (FM) system. We evaluate several different software implementation architectures for the remote agent, characterizing their differing performance characteristics. Our Put/Get FM implementation achieves peak bandwidths for put/get operations of 67 MBytes/s, overheads of a few microseconds, and remote read latencies as low as 26 microseconds on a Myrinet connected PC cluster. This implementation was released publicly as part of HPVM 1.0 in August 1997, and is receiving significant usage. It has been used for an implementation of the Global Arrays library and also serves as a back-end target for PGI´s commercial HPF compiler