Optimizing search for malware by hashing smaller amounts of data

The current information security threat landscape now includes advanced persistent attackers whose tactics, techniques, and procedures (TTPs) are designed to provide for a stealthy infiltration of systems, lateral movement, privilege escalation, and exfiltration of sensitive data (typically for purposes of corporate espionage). Once forensically detected, these same TTPs act as `fingerprints´ (called indicators), locating where the attackers have been elsewhere with a network of host computers. A very commonly used technique to locate compromised systems is to calculate the MD5 hash values for each file within a filesystem and compare against a known `bad fingerprint´ (hash value of known malware components). Our research demonstrates that of all of the indicators, the exhaustive computation of MD5 hash values across the full file contents of every file within the entire filesystem consumes the largest amount of time. Given the goal of reducing the time to determine if a system is compromised, we develop a novel approach to optimizing the creation of these indicators. Our experiments identify that computing a MD5 hash value of just the first four kilobytes (4KB) of each file provides a substantially faster method to search for specific MD5-based indicators with a very low false positive rate.