Estimating semantic confidence for spoken dialogue systems

In order for a spoken dialogue system to carry on a fluent conversation, it must be able to estimate confidence in its interpretation of the input that it is receiving. It must realize when it doesn´t understand the user and interact to correct the problem. To this end, most systems have some form of confidence assessment mechanism. These algorithms generally estimate confidence on a word-by-word basis and sometimes use these estimates to accept or reject an utterance as a whole. This paper presents the confidence assessment mechanism developed for the CU Communicator spoken dialogue system. The focus of this mechanism is to assess confidence in the semantic representation extracted from an utterance by the system rather than in the string of words produced by the recognizer. We use a decision tree classifier with features based on acoustic models, language models, word lattice density, parser output and dialogue context. The classification performance is evaluated on a test set from the CU Communicator data. CU Communicator is a telephone based spoken dialogue system for getting information on air travel, hotels and rental cars.