The fault in our scores: Impact analysis of distorted subjective data in objective QoE assessment

Subjective evaluation of service quality is considered by many to be the ultimate solution in the aspect of measurement accuracy, yet such methods consume immense resources and are difficult to automate in order to provide realtime service quality monitoring. More and more objective assessment techniques rise to efficiently estimate the level of perceived quality and a notable portion began involving subjective measurement data to reduce the gap between objective and subjective methods. However, a distorted subjective input may diminish accuracy instead of increasing it. The paper presents the analysis of distorted subjective evaluation utilized in automated assessment solutions. Four distinct usages of subjective data is investigated and shown how the distorted input may appear in method validation and quality estimation results.