Tracing long running applications: A case study using Gromacs

Performance analysis is inevitable to develop applications that utilize the enormous capabilities of current HPC systems. While many recent tool studies focused on large scales, performance analysis of long-running applications has not been paid much attention. This paper investigates challenges that arise from monitoring long-running real-life applications, in particular, the disruptive bias of intermediate memory buffer flushes in the measurement environment. We propose a concept for an in-memory event tracing that completely avoids intermediate memory buffer flushes. We evaluate to which extent such an in-memory event tracing workflow helps overcoming the critical properties, such as resulting trace size, application slow down, and measurement bias. We utilize a prototype implementation, based on Score-P and OTF2, with the molecular dynamics packages Gromacs, an application currently infeasible to monitor in a full production run.