Scalable Critical Path Analysis for Hybrid MPI-CUDA Applications

Utilizing accelerators in heterogeneous systems is an established approach for designing peta-scale applications. Today, CUDA offers a rich programming interface for GPU accelerators but requires developers to incorporate several layers of parallelism on both CPU and GPU. From this increasing program complexity emerges the need for sophisticated performance tools. This work contributes by analyzing hybrid MPI-CUDA programs for their critical path, a property proven to effectively identify application bottlenecks. We developed a tool which constructs a dependency graph based on an execution trace and the inherent dependencies of the programming models CUDA and MPI. Thereafter, it detects wait-states and attributes blame to responsible activities. Together with the property of being on the critical path we can identify activities that are most viable for optimization. The developed approach has been demonstrated with suitable examples to be both scalable and correct. Furthermore, we establish a new categorization of CUDA inefficiency patterns ensuing from the dependencies between CUDA activities.