Lessons learned from using the automated assessment tool “Mooshak”

Mooshak is an automatic online judge designed for evaluating algorithmic correction in programming contests, which has been also used in academia. In this paper we report preliminary results regarding an experiment involving the use of the tool in a course on algorithm design and analysis, for students in their second year of computer science degrees. The study was conducted in order to measure students´ general opinion about using Mooshak, and whether it could have positive effects on programming habits and other variables such as the drop-out rate. Results show that students did not embrace the experience in general, as seen through a reliable questionnaire that was constructed to evaluate students´ acceptance of Mooshak. Additionally, the online judge did not help to reduce the drop-out rate, as has been previously reported in the literature. Nevertheless, the use of Mooshak provided several benefits, such as a considerable increase in the percentage of students who use debugging tools, or a better reception of the online judge among a group of online students. Finally, several guidelines related to the lessons learned from the experiment are given in order to use Mooshak successfully for teaching.