Design and Analysis of a Parallel Data Distribution Overlay Topology

Cluster computers provide cost effective CPU and memory capacity, but many modern applications require extremely high aggregate I/O bandwidth to access large-scale, multidimensional data. Parallel file systems are one solution to this problem, but can be limited by various bottlenecks in the system. With the availability of large, cheap, fast disk drives, each cluster node can have its own copy of even large data sets, but copying the data to all the nodes becomes an impediment. Our approach uses a parallel virtual overlay topology to exploit the cluster´s network capacity to deliver a file to all nodes in roughly the same time it would cost to transfer it to a single node. The approach scales because the aggregate performance grows as the cluster size increase. In this paper, we evaluate the performance of our parallel data distribution architecture, exploring scalability, topology options, and bottlenecks. Based on the analysis, we developed an analytical approach to optimizing the topology based on system constraints.