Issues in the evaluation of spoken dialogue systems using objective and subjective measures

The paper presents results and conclusions about the current evaluation methodologies for spoken dialogue systems (SDS). The PARADISE paradigm, used for evaluation in the DARPA Communicator project, is briefly introduced and discussed through the application to the OVID home banking dialogue system. It is shown to provide results consistent with those obtained by the DARPA community, but a number of problems and limitations are pointed out. The issue of user attitude measures obtained through questionnaires is discussed. This is an area that has not received much attention from the speech technology community, but is important in order to obtain valid results and conclusions about usability. A general presentation of the issues that must be addressed when developing and employing questionnaires is given with a focus on how to ensure the reliability and validity of the results. Examples of results obtained from the OVID project are used to illustrate this.