A Low-level SDL-based Framework for Efficient Executions of Large-scale Scientific Workflows

Traditionally, high-performance computing has a focus on computing performance. With the growing scales of computing power, storage capacity, intra- and inter-cluster communication rate and the advent of technologies, researchers are building complex applications with large data sets to execute scientific workflows on distributed resources, so called "large-scale workflows". The general tendency shows that the amounts of data in such complex workflows may increase essentially. The storage limitations and the expensive I/O operators lead to develop an efficient and scalable technique to help researchers to execute large-scale workflows on HPC systems. In this paper we present a mechanism that provides highly efficient and scalable communications between distributed inter-dependent tasks within workflows. Our system is a simple and reliable framework to relieve the researchers from data management and network programming. Another major advantage of our approach is that, it supports various applications, from in-house codes to commercial software, without modification to the existing applications. Moreover, it provides direct and fully automatic data transferring among distributed tasks. Experimental results on a bio-mechanical workflow confirm that the framework has the potential to greatly improve performance and scalability.