Industrial Evaluation of a Log File Analysis Methodology

Test result evaluation programs often take the form of log file analyzers, which analyze text logs of events that have happened during testing. Previously, we proposed a methodology for deriving logging instrumentation and state-based log file analyzer programs from requirements. In this paper, we report on an industrial evaluation in which the methodology was carried out on two pieces of commercial software in a commercial setting. We report on our quantitative and qualitative observations and on recommendations for improving the methodology and the tools used.