MyCloud: On-Demand Virtual Cluster Provisioning on HPC Resources

Advances in the industry are gaining attentions from the scientific computing community. Programming models and frameworks such as MapReduce and Hadoop are attracting scientists due to their easy-to-use interfaces and autonomic parallel processing abilities. Compared with the conventional HPC execution environments, clouds possess desirable features such as administrative privilege, customizable software environments, and are more capable of supporting new frameworks and boosting inter-organization collaborations. Isolated private clouds and public clouds are used by scientists. However, extra administrative and monetary costs are inevitable in both cases. In this paper, we propose My Cloud, a resource management framework integrating cloud-based techniques into HPC systems. My Cloud supports both conventional HPC jobs and cloud-like on-demand virtual cluster provisioning on the same HPC cluster simultaneously. Dynamic resource sharing between the two environments is beneficial for cluster utilization and burst usages handling. The design of My Cloud is friendly to the state-of-the-art Software Defined Networking technologies and opens opportunities for fine-grained application-aware network engineering.