Computer supported evaluation in cooperative spoken dialog systems´ design projects

The paper examines the role of evaluation in collaborative work in design. It includes, on the one hand, to study the variations of evaluation needs along the development lifecycles. On the other hand, it demands to address the consequent adjustments of evaluation tools (metrics and dashboards) developed both locally and at the project monitoring level. Evaluation is also affected by human factors and issues related to cooperative work. As a case study, we approach the multidisciplinary field of SDSs to apprehend how CSCW tools could support the evaluation process in cross-disciplinary projects. We first review existing tools that implement design conventions for evaluators. They permit a convergence of evaluation practices while considering the various analytical points of view. Then, we present a framework based on the mapping between prototypical evaluation situations and relative evaluation templates. Finally, we propose a new framework that makes evaluation situations and evaluation schemes explicit so as to nurture a reflection on the heuristics that guide the evaluation design.