Matryoshka: Tunneled packets breaking the rules

Intrusion detection and prevention systems (IDPSs) are widely used to secure computer networks. They monitor network traffic by searching for unusual combinations in protocol headers and for malicious patterns in the packet payloads. In this paper we present "Matryoshka", a vulnerability that allows tunneled malicious packets to bypass the signature mapping procedures implemented in many industrial IDPS. Matryoshka is implemented as a tool and tested against Snort under different topologies and modes. To mitigate attacks that can be initialized using the bypassed tunneled malicious packets, a Snort preprocessor was developed and tested, and results demonstrated that all malicious tunneled packets were successfully detected. Processing overhead of the preprocessor to inspect and decapsulate tunneled packets was measured at 2% of the overall overhead of inspecting, decapsulating, and matching the malicious signature, and at 0:2% of the overall overhead of inspecting, decapsulating, assembling, and matching the signature.