A study of the effectiveness of two threshold definition techniques

Background: Measurement is a technique that is widely-used to quantify quality of process models. Evaluation of measurement results implies comparison against limit values, called thresholds. Determining thresholds is no trivial task and it requires the application of complex techniques. There are several techniques that have been published to date, proposing different approaches for threshold extraction. Two of the most prominent techniques are ROC curves and the Bender method. Although they come from different fields, both use logistic regression analysis as a discriminator function. Aim: For this reason, the main hypothesis is that thresholds obtained by both of those techniques are equally efficient in classifying the measurement results. Method: To check the hypothesis, we obtained thresholds for a group of empirically-validated measures for business process models, by applying both techniques. Then we checked the accuracy of the results. Results: The results indicate that the hypothesis should be rejected. Conclusions: ROC curves obtained more accurate thresholds for measurement evaluation.