A novel approach of detecting Trojan based on network behavior analysis

Most existing approaches for detecting Trojan are limited for obfuscation and encryption techniques. In this paper, we present a network behavior analysis designed to address the limitations of previously-proposed approaches. Our solution considered not only transport layer characteristics but also network layer characteristics. The approach in this paper exhibits two major advantages: (1) can better represent Trojan network behavior, and (2) performed at very low computational cost. Based on clustering technique, we proposed a detection model that detects Trojan communication with high accuracy. We implement the model on real-world traces. The experiments show that our model is suitable for detecting Trojan communication amongst the vast amount of network traffic, with over 90% accuracy and less than 3.5% false positive rate. We confidently consider that our detection approach is complementary to the existing techniques.