A Smart Fuzzing Method for Detecting Heap-Based Buffer Overflow in Executable Codes

This paper presents a new concolic execution-based smart fuzzer for detecting heap-based buffer overflow in the executable codes. The proposed fuzzer executes the target program with concrete input data and calculates the constraints of the executed path symbolically. The path constraints are used to generate test data that traverse new execution paths in the target program. For each executed path, the fuzzer also calculates heap-based buffer overflow constraints. These constraints determine what input data may cause heap-based buffer overflow in the executed path. By combining the path and vulnerability constraints new test data are generated, if possible, that traverse a specific execution path and activate specific vulnerability in that path. We implemented the proposed smart fuzzer as a plug-in for Valgrind framework. The implemented fuzzer is tested on different groups of test programs. The experiments demonstrate that the fuzzer can detect the vulnerabilities in these programs accurately.