Hybrid Embarrassingly Parallel algorithm for heterogeneous CPU/GPU clusters

High Performance Computing is focusing on heterogeneous architecture. The Embarrassingly Parallel algorithm is typical of Monte Carlo method which are widely applied to many important scientific areas. In this paper, we present an efficient Hybrid Embarrassingly Parallel algorithm for heterogeneous CPU/GPU clusters and an effective task distribution model for the load balancing between CPU and GPU. Our Hybrid EP algorithm can use the computing capability of both multi-core CPU and many-core GPU simultaneously based on the task distribution model. We test Hybrid EP algorithm on various types of CPUs, GPUs and the Tianhe-1A supercomputer. The overall performance speedup of M2050 GPU ranges from 10.84 times compared with six cores X5670 to over 50.53 times compared with quad cores Q6600. The performance of heterogeneous CPU/GPU Tianhe-1A supercomputer, in which both CPU and GPU are sufficiently used, outperforms pure CPU cluster 6.86 times. The speedup increases linearly with the number of nodes and the average efficiency is up to 98.72% for 4096 nodes.