Using concept maps for evaluating program objectives

In assessing students\´ conceptual understanding of their engineering discipline, traditional methods such as exams and surveys do not capture the level of knowledge integration that students have obtained about their field of study. At the University of Pittsburgh, we are investigating the use of concept maps to measure this "knowledge integration" as part of meeting program objectives. Unfortunately the qualitative nature of concept maps makes scoring and data analysis laborious. To combat this, a holistic metric was developed that effectively measures the comprehensiveness, structure and correctness of students\´ maps. In the spring of 2000, a sample of sophomore, junior and senior industrial engineering students developed concept maps of their field of study. This experiment was repeated in the fall of 2002 with seniors, many of who had participated in the initial experiment as sophomores. When traditional methods of scoring the maps yielded little information of value, a holistic approach was developed in an effort to quantify students\´ conceptual understanding of their engineering academic program. This paper describes the development of the holistic metric and analyzes the scored maps relative to a number of student related factors. Also discussed is how the scored maps can be further used to evaluate programmatic objectives.