Parallel Parameter Tuning for Applications with Performance Variability

In this paper, we present parallel on-line optimization algorithms for parameter tuning of parallel programs. We employ direct search algorithms that update parameters based on real-time performance measurements. We discuss the impact of performance variability on the accuracy and efficiency of the optimization algorithms and proposed modified versions of the direct search algorithms to cope with it. The modified version uses multiple samples instead of single sample to estimate the performance more accurately. We present preliminary results that the performance variability of applications on clusters is heavy tailed. Finally, we studay and demonstrate the performance of the proposed algorithms for real scientific application.