Automating the performance deviation analysis for multiple system releases: An evolutionary study

This paper presents a scenario-based approach for the evaluation of the quality attribute of performance, measured in terms of execution time (response time). The approach is implemented by a framework that uses dynamic analysis and repository mining techniques to provide an automated way for revealing potential sources of performance degradation of scenarios between releases of a software system. The approach defines four phases: (i) preparation — choosing the scenarios and preparing the target releases; (ii) dynamic analysis — determining the performance of scenarios and methods by calculating their execution time; (iii) degradation analysis — processing and comparing the results of the dynamic analysis for different releases; and (iv) repository mining — identifying development issues and commits associated with performance deviation. The paper also describes an evolutionary study of applying the approach to multiple releases of the Netty, Wicket and Jetty frameworks. The study analyzed seven releases of each system and addressed a total of 57 scenarios. Overall, we have found 14 scenarios with significant performance deviation for Netty, 13 for Wicket, and 9 for Jetty, almost all of which could be attributed to a source code change. We also discuss feedback obtained from eight developers of Netty, Wicket and Jetty as result of a questionnaire.