Scaling HPC Clusters with volunteer computing for data intensive applications

Due to the computational demand of data intensive applications, parallel computer hardware such as the HPC Cluster system is required to execute such the applications. However, building large HPC Clusters for this sole purpose is not always feasible or even not cost-effective since the purchasing, operational and maintenance cost of´ the dedicated systems is too high but they are not fully utilized in most of the time. In this regard, Volunteer Computing can address this problem as it provides a large amount of computing resources at no cost. We develop a system that expands the computing capability of HPC Clusters by using the additional computing power donated by volunteer users who would like to give the computing resources of their unused desktop computers to help execute jobs in the HPC Clusters. The proposed system can combine the native Cluster compute nodes and a set of´ non-dedicated compute nodes contributed by volunteers. The experiments demonstrate that the volunteer resources, such as CPU time, disk storage and GPU, can be seamlessly integrated into the HPC Clusters allowing the systems to dynamically scale up/down regarding to the amount of resources in Volunteer Computing.