Access-pattern and bandwidth aware file replication algorithm in a grid environment

Replication in grid file systems can significantly improve I/O performance of data-intensive grid applications, but its manual creation and placement would be impractical in a real grid environment involving thousands to millions of files accessed per application. Although automatic determination of where and how many replicas should be created should be decided with regards to application access patterns and network throughputs, thereby achieving high application throughput and minimizing replica space overhead, previous studies have focused on limited parameter spaces in their algorithmic optimizations. We propose an automated replication algorithm that allows most of I/O accesses to be performed within a given time threshold, while simultaneously minimizing the space overhead by replication. Our algorithm models the replication problem as a combinatorial optimization problem, where the constraints are derived from the given access time threshold and various system parameters, while the objective function being to minimize file replication costs. We solve the optimization problem by dynamically monitoring and estimating inter-node link throughputs and file access patterns of running applications. Our simulation-based studies suggest that the proposed algorithm can achieve higher performance than simple techniques, such as ones that always or never create replicas, while keeping storage usage very low. The results also indicate that the proposed automated algorithm can perform comparably with manual replica placement.