Automated evaluation of regular lab assignments: A bittersweet experience?

Our undergraduate curriculum includes a one semester course on object-oriented programming in the C# language and the .NET platform. The course underwent a major change in its labs organization two years ago - an introduction of regular one-week lab assignments motivated by a need to enhance unsatisfactory skills of students attending the course. However such change of the course scheme would not be possible without introduction of an automated evaluation system of the lab assignments. Besides the expected positive results, we were worried about any unforeseen consequences of the change. Having enough data from before and after the change of the course scheme the paper presents a unique analysis of impact of such a change on students and the course itself. The paper provides two main contributions: (1) to our best knowledge, we provide the first empirical validation of expected impacts on real-world data and (2) we identify and discuss several unexpected aspects and drawbacks of introduction of such a new scheme.