Big Data Techniques for Scalable In-Band and Out-of-Band HPC Energy Measurement

Research in high performance computing (HPC) energy optimization is a growing field motivated by cost and environmental drivers. As commodity server platforms are increasingly deployed as affordably scalable compute clusters, the processor and operating system´s energy management capabilities also continues to advance in sophistication. This trend creates a large number of configuration and control parameter combinations that can affect a parallel program´s performance and energy consumption. In pursuit of a systematic methodology for determining the optimal low-energy configuration, we have developed a precise CPU/DRAM energy measurement system that simultaneously records both out-of-band and in-band measurements for any given benchmark code executing on an entire or a defined sub-portion of an HPC compute cluster. The recording of high sample-rate, program-synchronized energy usage statistics across a multi-processor cluster from two independent measurement systems generates a large volume of experimental data. We also show how Big Data tools and techniques can make the analysis of such data sets manageable in processing the experimental output. The measurement framework and associated instrumentation are sufficiently scalable to support any program-level energy optimization research in HPC parallel systems.