A non-asymptotic standardization of binomial counts in Higher Criticism

In the “Higher Criticism” framework for multiple hypothesis testing, the binomial counts of the empirical CDF of the P-values are standardized through Z-scores. In the asymptotic regime, the distribution of these standardized counts tends to be a standard Gaussian. In the non-asymptotic regime, these distributions depart from normality and generally differ from each other. As a result, comparable values of Z-scores obtained for different P-values reflect in fact different evidence against the joint null hypothesis. We show that such evidence can be computed in closed-form for finite sample size, resulting in an alternate standardization which leads to identically distributed and thus more comparable statistics. Numerical results illustrate some merits of the alternate standardization.