PSYCHOMETRIC PROPERTIES OF A NEWLY DEVELOPED ANALYTIC RATING SCALE

In performance testing in general and in second language performance testing in particular, measurement instruments are used to evaluate and rate the students’ performances or products. Rating scales are the most commonly used instruments for this purpose. To produce consistent results, these instruments should be both reliable and valid—two critical characteristics of any measurement instrument. In this study, we developed a new rating scale. The scale we constructed was an analytic rating scale in which students’ essays were rated on a scale ranging from 1 to 6. The scale consisted of 15 items which eight raters used to rate thirty students’ essays. A many-facet Racsh measurement model (Linacre, 1994) was used to analyze the data. The results of Facets analysis showed that the scale functioned properly well. The findings suggest that we could use many-facet Rasch measurement to develop new rating scales. This study has some implications for rating scale developers, policy makers, and language teachers. Language teachers, for example, could use the new rating scale in this study to rate their students ‘products—essays or speech samples—more objectively and more accurately.