A fast URL lookup engine for content-aware multi-gigabit switches

Cluster-based servers are one of the best solutions to build high-performance, scalable, and reliable Internet Web servers. A number of researches have been done about enabling the dispatcher in cluster-based Web servers to route the users requests based on higher layer information, such as URLs. Hashing functions and tree structure are often used to achieve the goal of URL lookup, but they may cause the problem of collision and result in unacceptable performance. This paper presents a fast scalable URL lookup mechanism that uses content addressable memory (CAM) as the basic hardware components. Our scheme not only supports exact matching of URL lookup, but also provides prefix-matching lookup ability so that it is very practically for URL content-filtering like systems. The proposed scheme takes constant time to lookup a URL and furnishes a rate of 100 million lookups per second. By applying the entry reuse concept, the expensive CAM space can be used in a more efficient way to store more URLs. With this fast URL lookup engine, the performance of content dispatchers or URL content filters can be greatly improved.