Processing HDF5 Datasets on Multi-core Architectures

In order to make scientific middleware and applications more scalable, there is a need to design them in such a way that they can utilize the evolving multi-core processor architectures available in grid and cloud computing environments. In this paper, we analyze various processing and scheduling techniques on multi-core architectures based on scientific data characteristics and access patterns. More specifically, we conduct fine-grained analysis of scientific datasets such as HDF5 to make effective processing and scheduling decisions in multi-threaded programming. We present performance analysis on how processing threads can be scheduled on multi-core nodes to enhance the performance of scientific applications that process HDF5 data. To accomplish this we introduce a dynamic marking scheme to keep track of the progress of threads on each core. This can be used to help determine work allocation, which results in a decrease in overall application execution time.