Supporting a Light-Weight Data Management Layer over HDF5

Scientific simulations are now being performed at finer temporal and spatial scales, leading to an explosion of the output data, and challenges in storing, managing, disseminating, analyzing, and visualizing these datasets. Tools commonly used today for disseminating and visualizing such data have inherent limitations, making it extremely hard to deal with larger datasets. We have developed a light-weight data management tool, which allows server-side sub setting and aggregation on scientific datasets stored in HDF5, one of the most popular scientific data formats. To support a variety of queries efficiently, our tool generates code for hyper slab selector and content-based filtering, and parallelizes selection and aggregation queries efficiently using novel algorithms. Additionally, our tool also supports certain most recent HDF5 features including dimension scale and compound data type. Through extensive evaluation, we show that our system is capable of efficiently supporting a variety of queries, scaling performance by parallelizing the queries, and reducing wide area data transfers through server-side data aggregation. We demonstrate that even for sub setting queries that are directly supported in OPeNDAP, a tool widely used by data dissemination portals, the sequential performance of our system is better.