Affect as Information about Users´ Attitudes to Conversational Agents

This paper presents a novel method for automatic evaluation of conversational agents. In the method, information about userspsila attitudes and sentiments to conversational agents and their performance are achieved by analyzing their general emotional engagement in the conversation and specific affective states, and interpreting them using psychological reasoning of affect-as-information. In the evaluation experiment the userspsila attitudes to two Japanese-speaking conversational agents were checked simultaneously in a survey and using a system constructed on the proposed method. The results returned by the system revealed similar tendencies as the survey. Therefore the method is applicable as a mean of evaluation for Japanese-speaking conversational agents.