An experiment on comparing textual vs. visual industrial methods for security risk assessment

Many security risk assessment methods have been proposed both from academia and industry. However, little empirical evaluation has been done to investigate how these methods are effective in practice. In this paper we report a controlled experiment that we conducted to compare the effectiveness and participants´ perception of visual versus textual methods for security risk assessment used in industry. As instances of the methods we selected CORAS, a method by SINTEF used to provide security risk assessment consulting services, and SecRAM, a method by EUROCONTROL used to conduct security risk assessment within air traffic management. The experiment involved 29 MSc students who applied both methods to an application scenario from Smart Grid domain. The dependent variables were effectiveness of the methods measured as number of specific threats and security controls identified, and perception of the methods measured through post-task questionnaires based on the Technology Acceptance Model. The experiment shows that while there is no difference in the actual effectiveness of the two methods, the visual method is better perceived by the participants.