The evaluation of text quality: expert-focused and reader-focused methods compared

The authors compare a reader-focused text evaluation with an expert-focused evaluation by technical writers and subject/audience experts. The experts were asked to predict the problems readers had signaled in a government brochure about alcohol. On average, they predicted less than 15% of the reader problems and produced a lot of new problem detections. In addition, the experts showed little mutual agreement in their problem detections. Their results suggest that a reader-focused evaluation should not be substituted for an expert-focused evaluation. The paper ends with a discussion of methodological issues for this type of research