Using Control Charts for Detecting and Understanding Performance Regressions in Large Software

Load testing is a very important step in testing of large-scale software systems. For example, studies found that users are likely to abandon an online transaction if the web application fails to response within eight seconds. Performance load tests ensure that performance counters such as response time stays in the acceptable range after each change to the code. Analyzing load tests results to detect performance regression is very time consuming due to the large amount of performance counters. In this thesis, we propose approaches that use control charts, a statistical process control technique, to assist performance engineers in identifying test runs with performance regressions, pinpointing the components which cause the regressions, and determining the causes of regressions in load tests. Using our approaches, engineers will save time in analyzing the results of load tests.