Evaluating metrics for automatic mind map assessment in various classes

Over the past three years, we have been studying how automated evaluation of student mind maps (when compared to an expert map) shows student learning for a variety of metrics. The goal of this work is to build a system that would then allow students to evaluate their understanding of the terminology in a respective field. The weakness of our studies, so far, is that our focus group to study these metrics consists of a single course in the field of Computer Engineering, and though this class has been used over multiple years to demonstrate the feasibility of our approach in a longitudinal study, a broader study needs to be done. In this work, we show how our current metrics perform across three additional fields; specifically, we have collaborated with instructors in speech pathology, communications, and political science (in addition to our traditional class in computer engineering). We then use our methodology to determine if these courses and a term long mind map exercise have similar results than previously reported and are these results evidence of student learning. Our results show that our existing metrics have similar results for one of the three new courses. However, in the two other courses, the data shows no evidence of learning based on the mind map exercise. Each of the instructors of these courses describes their experience with the activity. Additionally, we evaluate the construction of the expert maps in each course to understand if there is a graph-based structural reason why we the results might be different. We conclude by suggesting our methodology is good for courses where terminology is clearly defined and is used and studied throughout the semester, and describe some future directions for this research.