Affect-as-Information Approach to a Sentiment Analysis Based Evaluation of Conversational Agents

In this paper we propose a novel method for automatic evaluation of conversational agents. The method is based on analyzing the userpsilas affect conveyed in utterances. From analyzing: the userpsilas general emotional engagement in the conversation and the emotion types conveyed by the user in the conversation, a simple psychological reasoning is derived about the userpsilas sentiment about the agentpsilas performance. The evaluation experiment on two Japanese-speaking conversational agents showed the same tendencies in the results returned by the system constructed on the proposed method and the userpsilas opinion about the two agents checked in the afterward survey. Thus the method can be used for evaluation of Japanese-speaking conversational agents.