مرکز منطقه ای اطلاع رساني علوم و فناوري - RELIABILITY OF RATERS FOR WRITING ASSESSMENT: ANALYTIC - HOLISTIC, ANALYTIC - ANALYTIC, HOLISTIC

Abstract :

One of the main concerns in writing assessment is the choice of reliable and valid rating criteria to decide on students’ writing proficiency levels. For this purpose, holistic and analytic rubrics have been employed most commonly by EFL/ESL programs and specialists as essay scoring instruments. In this present study, based on their actual use of rubric types, 31 novice Turkish teachers of English who were responsible for rating 344 student essays were randomly appointed as either holistic raters or analytic raters. Given that there were two rater pairs per essay, three different conditions and approaches to essay scoring were realized and these were used for this correlational study: holistic versus holistic, holistic versus analytic, and analytic versus analytic. Interrater reliability was determined through the study of these three types of scoring conditions to find the amount of correlation between raters’ scores. Results determined that the highest correlation occurred most strongly between two holistic raters followed respectively by two analytic raters. The study also revealed that inter-rater reliability is rather low in a condition when two different rater types – holistic versus analytic – score the same student essay.