Evaluating complex scales through subjective ranking

In this paper, a subjective assessment methodology based on ranking, reordering and image categorization is proposed. It is designed to facilitate the evaluation of complex features such as monocular depth cues in natural images. These features can be particularly challenging to explain to test participants, and the proposed methodology is designed to provide to the test participants more insights and examples about the scale under evaluation. The performances of the method are compared to the absolute category rating methodology, and the advantages of each part of the test design are studied to verify the added value of the different steps of the evaluation procedure. Results show that the method is promising and can improve stability between subjective test participant results.