Acoustic correlates of user response to error in human-computer dialogues

Using tagged data from the DARPA Communicator Project, we investigate acoustic features of user responses to system errors. We measure acoustic parameters such as energy, fundamental frequency, sub-band energy, ratios of voiced, unvoiced and silent regions of speech, fundamental frequency slope, spectral slope, and spectral center of gravity. We investigate different types of user responses to the errors, including frustration and various types of corrections. It is confirmed that the most prominent acoustic parameter for responses to the errors is fundamental frequency maximum and range, while other features are found to be salient for specific reaction types. More interestingly, acoustic characteristics of user responses to the errors are found to be different depending on whether the responses are the initial or continued responses to the errors. Similarly, normal user responses can differ acoustically depending on whether or not they were preceded by responses to error. We also present results on automatic classification of error response types using these features.