Empirical evaluation of interactive multimodal error correction

The first commercial dictation systems for continuous speech have become available. Although they generally received positive reviews, error correction is still limited to choosing from a list of alternatives, speaking again or typing. We developed a set of multimodal interactive correction methods which allow the user to switch modality between continuous speech, spelling, handwriting and pen gestures. We integrated these correction methods with our large vocabulary speech recognition system to build a prototypical multimodal listening typewriter. We designed an experiment to empirically evaluate the efficiency of different error correction methods. The experiment compares multimodal correction with methods available in current speech recognition applications. We confirm the hypothesis that switching modality can significantly expedite corrections. However in applications where a keyboard is acceptable, typing correction remains the fastest method to correct errors for users with good typing skills. If the keyboard is not desired, either due to application constraints or user preferences, our multimodal error correction enables state-of-the-art speech recognition technology to deliver keyboard-free text input which beats fast unskilled typing in input speed, including the time necessary to correct errors