Building a Test Suite for Web Application Scanners

This paper describes the design of a test suite for thorough evaluation of web application scanners. Web application scanners are automated, black-box testing tools that examine web applications for security vulnerabilities. For several common vulnerability types, we classify defense mechanisms that can be implemented to prevent corresponding attacks. We combine the defense mechanisms into "levels of defense" of increasing strength. This approach allows us to develop an extensive test suite that can be easily configured to switch on and off vulnerability types and select a level of defense. We evaluate the test suite experimentally using several web application scanners, both open-source and proprietary. The experiments suggest that the test suite is effective at distinguishing the tools based on their vulnerability detection rate; in addition, its use can suggest areas for tool improvement.