A Network Performance Based Data Placement Policy in Distributed Data-Intensive Applications

Data Placement Policy is a key factor to the performance of distributed data-intensive applications. Usually placing massive data sets efficiently can improve user experience of data-intensive scientific applications under large scale distributed environments. In distributed file systems and distributed storage systems, it focused on the number of redundant data to keep data available all the time. In this paper, we improve the performance of applications by making the correlation of pure network performance measurements with the end-to-end data placement performance. By modeling massive data and adjusting network measurement system, a network performance based data placement policy is introduced. Under this policy, we can make the massive data not only more available, but also efficient.