A novel software-based MD5 checksum lookup scheme for anti-virus systems

In recent years, the size of virus signature databases has been growing rapidly, leading to a corresponding reduction in the performance of anti-virus (AV) software. In general, virus signature databases comprise string-based and hash-based (e.g., MD5) signatures. Currently the majority of signatures are hash-based and Cloud-based AV systems rely on them as the local cache to reduce the network loading. In this paper, we provide a novel scheme for looking up MD5 checksums to improve virus scanning performance involving hash-based signatures. The authors treat the range hash in which characters occur as a filter to avoid unnecessary lookups and keep the range of the exact search range to a minimum. The scheme is 135 times faster than ClamAV´s in clean/general cases and only required 4MB of memory for hash-based filtering. This scheme could easily be extended to other hash-based applications.