Non-intrusive process-based monitoring system to mitigate and prevent VM vulnerability explorations

Cloud is gaining momentum but its true potential is hampered by the security concerns it has raised. Having vulnerable virtual machines in a virtualized environment is one such concern. Vulnerable virtual machines are an easy target and existence of such weak nodes in a network jeopardizes its entire security structure. Resource sharing nature of cloud favors the attacker, in that, compromised machines can be used to launch further devastating attacks. First line of defense in such case is to prevent vulnerabilities of a cloud network from being compromised and if not, to prevent propagation of the attack. To create this line of defense, we propose a hybrid intrusion detection framework to detect vulnerabilities, attacks, and their carriers, i.e. malicious processes in the virtual network and virtual machines. This framework is built on attack graph based analytical models, VMM-based malicious process detection, and reconfigurable virtual network-based countermeasures. The proposed framework leverages Software Defined Networking to build a monitor and control plane over distributed programmable virtual switches in order to significantly improve the attack detection and mitigate the attack consequences. The system and security evaluations demonstrate the efficiency and effectiveness of the proposed solution.