On reducing the processing load of Redundancy Elimination algorithms

Redundancy Elimination (RE) algorithms, which typically are used for improving network performance, work by splitting network packet payload in content dependent chunks and comparing their fingerprints for similarity. However, due to excessive processing power demand, most of the proposed algorithms are not suitable for real world deployment. This work proposes a novel technique, which takes into account the traffic type (video, text, music, etc.), and uses that to reduce the processing load form the RE boxes. Preliminary results show that this technique decreases the processor load by an average of 75% with an example mix of text and binary files, while keeping the compression factor comparable to that of similar algorithms. The total memory access, and the cache size per node are also decreased greatly.