Enhancing security using legality assertions

Buffer overflows have been the most common form of security vulnerability in the past decade. A number of techniques have been proposed to address such attacks. Some are limited to protecting the return address on the stack; others are more general, but have undesirable properties such as large overhead and false warnings. The approach described in this paper uses legality assertions, source code assertions inserted before each subscript and pointer dereference that explicitly check that the referencing expression actually specifies a location within the array or object pointed at run time. A transformation system is developed to analyze a program and annotate it with appropriate assertions automatically. This approach detects buffer vulnerabilities in both stack and heap memory as well as potential buffer overflows in library functions. Runtime checking through using automatically inferred assertions considerably enhances the accuracy and efficiency of buffer overflow detection. A number of example buffer overflow-exploiting C programs are used to demonstrate the effectiveness of this approach.