Assuring fault classification agreement - an empirical evaluation

Inter-rater agreement is a well-known challenge and is a key issue when discussing fault classification. Fault classification is, by nature, a subjective task since it highly depends on the people performing the classification. Measures are required to hinder the subjective nature of fault classification to propagate through the fault classification process and onto subsequent activities using the classified faults, for example process improvement. One approach to prevent the subjective nature of fault classification is to use multiple raters and measure inter-rater agreement. We evaluate the possibility to have an independent group of people classifying faults. The objective is to evaluate whether such a group could be used in a process improvement initiative. An empirical study is conducted with eight persons classifying 30 faults independently. The study concludes that the provided material were unsatisfactory to obtain inter-rater agreement.