Writing with speech: A qualitative user evaluation study

This paper explores qualitative aspects of evaluating automatic speech recognition (ASR) based dictation. We propose an evaluation method of ASR-based dictation from the point of view of the user. To gather data for the study, we conducted experiments with possible target users (n = 6), asking them to compose the same text using ASR-based dictation software for the Hungarian language, and to repeat the same task using three different ASR systems. An inherent part of this task is correcting errors made by the system. We found that users who had less experience with ASR, are biased towards keyboard & mouse writing mode, the one that they were more familiar with. Moreover, they overestimate the duration of text production using speech (ASR-based dictation) as a writing mode, when compared to keyboard & mouse writing mode. As expected, higher number of errors in the ASR output lead to longer error correction time, but somewhat surprisingly, longer edit time is in negative correlation with the number of errors left in the corrected text. That is, even if the time taken for correcting ASR errors was longer, the quality of the corrected text was worse than the one of the system with low recognition errors.