The Impacts of Supporting Materials on Code Reading: A Controlled Experiment

Background: Code inspection has been accepted as an effective method to detect and remove defects and code reading is a critical step in code inspection. However, there are very limited empirical studies on the content and appropriate forms of the suitable software artifacts as the supporting materials, hence inspectors may not be well-supported with necessary knowledge to carry out code reading. Objective: This research aims to investigate the impact of different common supporting materials (i.e., comments vs. design documents) on code reading. Method: A relatively large-scale controlled experiment with 135 senior students was designed and executed to compare the impacts of different supporting materials on code reading. The subjects were randomly separated into three groups with different treatments, i.e, the comments, the design documents and the comments+design documents, respectively. Two metrics regarding the code reading performance (i.e., Effectiveness and Defect Detection Rate) were used to compare the different impacts derived from the two different types of supporting materials. Qualitative feedbacks were also collected using questionnaires for the final analysis. Results: The results indicate that students performed better when being provided with comments than comments+design documents. Also, the removal of design documents shows little impact on inspection effectiveness and may lead to an increase in defect detection rate. Conclusion: Comments may provide more help and value than design documents as supporting material in small to median sized code reading.