A strategy to jointly test image quality estimators subjectively

We present an automated algorithm to design subjective tests that have a high likelihood of finding misclassification errors in many image quality estimators (QEs). In our algorithm, a collection of existing QEs collaboratively determine the best pairs of images that will test the accuracy of each individual QE.We demonstrate that the resulting subjective test provides valuable information regarding the accuracy of the cooperating QEs. The proposed strategy is particularly useful for comparing efficacy of QEs across multiple distortion types and multiple reference images.