Investigating a scenario-based performance assessment of engineering professional skills

Professional skills are critical for success in the multidisciplinary, intercultural, and cross-functional team interactions that characterize 21st century engineering careers. Despite consensus around the need for developing professional skills, engineering programs have been struggling to define, teach, and measure those skills. This paper describes the investigation of the functioning of a group-based performance assessment of engineering professional skills called EPSA. The EPSA consists of a scenario that presents a contemporary engineering issue with no clear-cut solution, and an analytical scoring rubric with five dimensions corresponding to the skills identified as 3f - 3j in ABET Criterion 3 Student Outcomes. This investigation dealt with two persistent problems in performance assessment: the extent to which student performance varied unexpectedly as a function of the assigned scenario, and whether complex performances could be disaggregated through the use of rubric dimension scores. Data were collected from 20 discussion groups at three engineering colleges in the US. The analyses were conducted using a two-stage nested mixed-effects model. Because of the small sample size, results are best viewed as preliminary and in need of replication. The findings tentatively suggest that EPSA scores discriminate among student groups and that differences in group performance related to scenario appear to be minimal. Further, student groups exhibited different proficiency profiles across the five EPSA dimensions. Results may suggest the need for more emphasis on engineering professional skills at the undergraduate level.