DDSN: Duplicate detection to reduce both storage and bandwidth consumption

As highly centralized storage facilities are gaining popularity, duplicate detection becomes a critical problem. Traditional methods focus on reducing the storage space consumption; however, for network storage system with remote clients, the network overhead cannot be ignored, especially when the system is accessed over WAN. We propose a new duplicate detection method and implement a network file system prototype called DDSN based on this new method. It can reach the same performance in terms of storage space consumption as the state-of-the-art sliding blocking method. Meanwhile, our method overcomes its drawback that the whole file needs to be transmitted over the network, and therefore saves massive bandwidth for duplicate data. Experiments confirm the effectiveness of the proposed method.