An Evaluation of Diagnosis in a Learning Environment for Object-Oriented Modeling

This paper reports the first evaluation of a diagnostic module, called ACDC (Automatic Class Diagrams Comparator) since its integration into Diagram, a learning environment for UML class diagrams. ACDC is an hybrid matcher that compares the student diagram with a reference diagram and tries to match them. It produces a list of structural and content differences between these two diagrams. These differences are then analysed by the pedagogical component of Diagram for producing appropriate feedbacks. The evaluation of ACDC system was conducted using diagnosis traces and student diagrams collected in a Diagram experimentation during Fall 2008 with students in a UML course. For evaluating the quality of ACDC, we use four match quality measures: Precision, Recall, F-Measure and Overall. The evaluation results show that the diagnosis are fairly well but requires to be improved in some specific situations.