Guidelines for Benchmarking Automated Software Traceability Techniques

To comparatively evaluate automated trace ability solutions, we need to develop standardized benchmarks. However there is currently no consensus on how a benchmark should be constructed and used to evaluate competing techniques. In this paper we discuss recurring problems in evaluating trace ability techniques, identify essential properties that evaluation methods should possess, and provide guidelines for benchmarking software trace ability techniques. We illustrate the properties and guidelines using empirical evaluation of three software trace ability techniques on nine data sets.