An analysis of the area under the ROC curve and its use as a metric for comparing clinical scorecards

There are many examples in the literature of scorecards derived from clinical data. These scorecards are proposed for use by health professionals to stratify patients into risk categories and are often compared using receiver operating characteristic (ROC) curves and their associated areas (AUC). This paper analyses random scorecards and shows that the underlying distributions and therefore statistical significance of the AUC metric is dependent on both the dimensionality of the scorecard and the distribution of classes in the data. This finding suggests that scorecards of differing dimensionality and distribution should not be compared directly on AuC values and that smaller scorecards would be expected to have a lower mean solely by chance.