Unanswerable queries in a spontaneous speech task

This paper outlines the general strategies followed in developing the CMU (Carnegie Mellon University) speech understanding system. Our system is oriented toward the extraction of information relevant to a task. It uses a flexible frame-based parser. Our system handles phenomena that are natural in spontaneous speech, for example, restarts, repeats and grammatically ill-formed utterances. It maintains a history of the key features of the dialogue. It can resolve elliptical, anaphoric and other indirect references. In general, the system may not be able to generate a database response to a query for a number of reasons, for example, the user may inquire about information not in the database. These unanswerable (Class X) queries constituted about 30% of the data in the November 92 ARPA Spoken Language Systems benchmark evaluation. We will present some results which indicate that the perplexity of the unanswerable queries is significantly higher than the perplexity of the remaining data. In this paper, we focus on our attempts to improve the performance of CMU´s Spoken Language System on the unanswerable queries. The system has been used to model an air travel information service (ATIS) task. In the December 93 DARPA Spoken Language Systems benchmark evaluation, the CMU ATIS system correctly answered 90.7% transcript inputs and 86.8% speech inputs. These were the best numbers reported for the evaluation