Influence of Text and Participant Characteristics on Perceived and Actual Text Difficulty

Because patients customarily receive medical text that is difficult to understand, we are developing a simplification algorithm to support simpler writing by medical professionals. Our algorithm relies on term familiarity and automatically suggests alternative wordings from different sources. We conducted a user study (N=17) to evaluate its effectiveness on reducing perceived and actual difficulty. Perceived difficulty was measured using sentences and a Likert-scale. Actual difficulty was measured using documents and multiple-choice and Cloze tests. We found a strong significant simplification effect for perceived difficulty (p=.002), but no effect for actual difficulty: only 6.2% improvement on the Cloze test. Evaluating participant characteristics showed that reading more newspapers or magazines correlated with lower multiple-choice (r=-386, p=.016) and Cloze test (r=-.340, p=.025) scores. STOFHLA scores, a health literacy measure, correlated with the Cloze test scores (r=.461, p=.002).