Smart hashing based queries in the cloud

High system dependability and high disaster recovery are important requirements for cloud service providers. Large-scale cloud systems providers often employ remote backups to help satisfy these requirements. However, due to long-distance and limited network bandwidth, backups may take very long time to finish and may become very costly. This problem increases as the size of stored image files in the cloud increases rapidly. To achieve high backup performance, this paper proposes a smart hashing based scheme, called Smarter, to replicate files between long-distance data centers. Efficient queries are important to improve entire system performance. With the aid of efficient queries, the idea behind Smarter is to select the most representative (thus most valuable) files, as base files, for real-time backups, while leaving the remaining non-duplicate ones for offline (or off peak-hour) backups. Smarter leverages a suitable "division of labor" between the metadata and the contents of multimedia images to support cost-effective redundancy detection. By using the smart hashing, Smarter identifies representative files and removes identical files at the source system, and then computes the top-k similar files between the source and destination systems in order to obtain significant bandwidth savings and shortens the backup time. It thus efficiently manages the massive images that account for a large fraction of the data to be transferred during backups. We implemented Smarter and evaluated it in a real cloud backup system. Results demonstrate the superior efficiency.