Comparison of two mixed methods approaches for multimodal quality evaluations: Open Profiling of Quality and Conventional Profiling

To guide the practitioner´s work in choosing between assessment methods, defined criteria to compare their benefits, costs, and limitations are needed. The goal of this paper is two-fold. Firstly, we develop an extensive comparison model to guide between-method comparisons based on a literature review. The model is composed of four main criteria, called economy, excellence, implementation and assessment with a total of 24 sub-criteria. Secondly, we conduct a comparison study between two mixed methods in which a subset of criteria of the model is examined. We compare Open Profiling of Quality utilizing individuals´ own vocabulary and Conventional Profiling utilizing fixed vocabulary in their descriptive evaluation. The study is conducted with naïve participants with varying 3D video qualities on a mobile device. The results compare both methods on a subset of comparison criteria and show that operationalization of the developed comparison model can provide a tool for holistic methods comparison.