API Fuzz Testing for Security of Libraries in Windows Systems: From Faults To Vulnerabilites

Application programming interface (API) fuzz testing is used to insert unexpected data into the parameters of functions and to monitor for resulting program errors or exceptions in order to test the security of APIs. However, vulnerabilities through which a user cannot insert data into API parameters are not security threats, because attackers cannot exploit such vulnerabilities. In this paper, we propose a methodology that can automatically find paths between inputs of programs and faulty APIs. Where such paths exist, faults in APIs represent security threats. We call our methodology Automated Windows API Fuzz Testing II (AWAFTII). This method extends our previous research for performing API fuzz testing into the AWAFTII process. The AWAFTII process consists of finding faults using API fuzz testing, analyzing those faults, and searching for input data related to parameters of APIs with faults. We implemented a practical tool for AWAFTII and applied it to programs in the system folder of Windows XP SP2. Experimental results show that AWAFTII can detect paths between input of programs and APIs with faults.