Developing Auditory-Perceptual Judgment Reliability in Otolaryngology Residents

Objectives To determine how standard residency training and educational background affect otolaryngology resident auditory-perceptual judgments compared with inexperienced listeners. A secondary aim was to assess the impact of a brief training session on the reliability of judgments among otolaryngology residents. Study Design Mixed cross-sectional and cohort study. Methods Twenty adult dysphonic and four normal speaker samples were selected from a database. Fifteen listeners at various stages of residency in otolaryngology judged all samples for breathiness and roughness using visual analog scales. Fifteen inexperienced listeners judged the same samples. Subsequently, 12 otolaryngology residents underwent a brief training session. During training, listeners were provided definitions of rating dimensions and accuracy feedback while rating 20 novel (training) stimuli. The feedback included averaged responses from experienced clinicians who had previously evaluated the same speech samples. The residents then completed posttraining evaluations. Results As a group, otolaryngology resident listeners had significantly better interrater agreement for judgments of roughness compared with inexperienced listeners (Pآ <آ 0.05) and trended toward better interrater agreement for judgments of breathiness (Pآ =آ 0.058). Posttraining, no significant improvement in interrater agreement for judgments of roughness was observed; however, a significant improvement was found in interrater agreement of judgments of breathiness (Pآ <آ 0.05). Conclusions Compared with inexperienced (untrained) listeners, otolaryngology resident listeners had better interrater agreement overall for judgments of dysphonia. A short training module was associated with improved reliability in resident evaluation of breathiness. Results have implications for improving and standardizing resident evaluation of dysphonia.