Security investigation and enhancement of IKEV2 protocol

IPsec has become a very popular Internet security infrastructure today. As a new key exchange protocol of IPsec, to some extent, IKEv2 can use cookie negotiation mechanism to detect and resist memory-based denial-of-service (DoS) attack in the application layer. However, IKEv2 still cannot avoid IP fragment-based DoS attacks since the IKEv2 messages transmission runs over UDP and there are large IKE messages needed to be fragmented during the exchange process between two IKE peers. In this paper we first investigate some typical methods and give the analysis of their inability against the IP fragmentation DoS attack. To overcome this problem, we design a new IKEv2 header format called M-ISAKMP, and add a new type of Notification Payload and other related strategies. With the novel application-based fragmentation mechanism, our proposed enhanced IKEv2 protocol achieves defending against DoS attack successfully and efficiently.