Asynchronous Collective Output with Non-dedicated Cores

Parallel applications are evolving to place larger demands not just on computation and network capabilities, but on storage systems as well. Storage hardware has scaled to keep up, but the software to drive it must evolve alongside to service this increased potential. This paper presents an output forwarding middleware for message-driven parallel applications written in Charm++. This layer directs IO operations across the entire system to a designated subset of processors in order to minimize contention and overheads. Our implementation is distinctive in that these processors are not dedicated to this task, but can still contribute to the computational task. Other processors need not block while waiting for the designated IO processors to become ready or make progress. Using this new layer, we demonstrate speedups of 1.5 - 2.5× in the popular scientific code NAMD over its previous parallel output implementation, along with reduced sensitivity to IO subsystem parameters.