Evaluating interrater agreement with intraclass correlation coefficient in SPICE-based software process assessment

As software process assessment (SPA) involves a subjective procedure, its reliability is an important issue. Two types of reliability have intensively been investigated in SPA: internal consistency (internal reliability) and interrater agreement (external reliability). This study investigates interrater agreement. Cohen\´s Kappa coefficient has been a popular measure for estimating interrater agreement. However, the application of Kappa coefficient in certain situations is incorrect due to the "Kappa Paradoxes". To cope with the insufficiency of Kappa coefficient, this study applied the intraclass correlation coefficient (ICC) to estimate interrater agreement. The ICC has not been employed in the SPA context. In addition, we examined the stability of the estimated ICC value by using a bootstrap resampling method. Results show that ICC could be applied where the Kappa coefficient could not be applied, but not all cases.