Using Branch Correlation to Identify Infeasible Paths for Anomaly Detection

In this paper, we propose a system called infeasible path detection system (IPDS) to combat memory tampering attacks causing invalid program control flows. In our system, the compiler analyzes correlations between branches and then the analyzed information is conveyed to the runtime system. The runtime system detects dynamic infeasible program paths by combining compiler determined information with runtime information to check the legality of the path taken during execution. IPDS achieves a zero false positive rate and can detect a high percentage of memory tampering for many attacks in which the tampering actually causes a change in control flow. Moreover, IPDS only incurs a modest amount of hardware resource and negligible performance penalty