Constructs Emerging from Students’ Scores Using Five Assessment Instruments at the End of Pre-Clerkship Phase in a PBL Curriculum

Introduction: Developing a framework of educational outcomes which is aligned with student assessment helps in driving the student learning to achieve these outcomes. The validity of this framework is essential for ensuring fairness to learners and to society.Objective: The goal of this study is to investigate the validity of the underlying domains of the assessment framework emerging from the students’ scores using five instruments used at the end of a pre-clerkship, problem-based medical curriculum.Method: Medical students (n=245, Year 4) were enrolled in the study during two consecutive academic years (2011 and 2012). We examined the construct validity of students’ scores from the following assessment instruments: multiple choice questions (MCQs), integrated short answer questions (SAQs), objective structured clinical examinations (OSCE), real patient clinical examinations (PCE) and computer-based clinical simulations (CCS). An exploratory factor analysis (EFA) was carried out followed by confirmatory analysis (CFA) using structural equation modelling (SEM) to evaluate the constructs emerging from the students’ scores of the five instruments.Result: The analysis yielded four inter-related constructs, called medical knowledge, clinical skills, procedural skills, and reasoning skills. In addition, the four constructs loaded on a higher order construct (called competence) with high regression weights. Although the overall fitness of the separate assessment instruments was poor, the fitness indices of the three factor model after cross-loadings between variables of different constructs were improved ( (χ2 [94] = 149.2, p 0.01 (Bollen-Stine p value = 0.104), CFI = 0.96, TLI = 0.94 and RMSEA = 0.04 (95% CI = 0.03-0.07).Conclusion: The study indicates better validity evidence of the structure from the combined scores of the five instruments in explaining ‘clinical competence” than the individual assessment instruments. The study also demonstrates that computer-based case simulations measure a separate construct which is different from measuring clinical skills in real or standardized patients.