Subjective quality evaluations using crowdsourcing

This paper presents a system and methodology for crowdsourcing subjective quality evaluations. The goal of this work is to study the feasibility of fast, inexpensive, and reliable subjective quality evaluations with crowdsourcing. We present a web-based system for conducting subjective quality evaluations using crowdsourcing platform. The results were evaluated by comparison with the formal subjective tests conducted by MPEG as a part of video standardization process. High quality compressed videos with known MOS values are used as references in subjective evaluations instead of the original lossless videos to address the bandwidth limitations. The results show that crowdsourcing can be a reliable platform for subjective quality evaluations.