User-centered evaluation for machine translation of spoken language

In this paper, we discuss a user-centered method for qualitative comparison of machine translation systems based on rankings of system output. System ranking requires no reference transcript, which can be expensive to generate and difficult to define for spoken language input. Ranking can be performed by monolingual users with no training in machine translation evaluation. We present results of experiments ranking four Arabic-to-English and three Mandarin-to-English machine translation systems processing spoken language transcripts with word error rates of 20-30%.