An Exploratory Study of the Evolution of Communicated Information about the Execution of Large Software Systems

A great deal of research in software engineering focuses on understanding the dynamic nature of software systems. Such research makes use of automated instrumentation and profiling techniques after fact, i.e., without considering domain knowledge. In this paper, we turn our attention to another source of dynamic information, i.e., the Communicated Information (CI) about the execution of a software system. Major examples of CI are execution logs and system events. They are generated from statements that are inserted intentionally by domain experts (e.g., developers or administrators) to convey crucial points of interest. The accessibility and domain-driven nature of the CI make it a valuable source for studying the evolution of a software system. In a case study on one large open source and one industrial software system, we explore the concept of CI and its evolution by mining the execution logs of these systems. Our study illustrates the need for better trace ability techniques between CI and the Log Processing Apps that analyze the CI. In particular, we find that the CI changes at a rather high rate across versions, leading to fragile Log Processing Apps. 40% to 60% of these changes can be avoided and the impact of 15% to 50% of the changes can be controlled through the use of the robust analysis techniques by Log Processing Apps. We also find that Log Processing Apps that track implementation-level CI (e.g., performance analysis) are more fragile than Log Processing Apps that track domain-level CI (e.g., workload modeling), because the implementation-level CI is often short-lived.