Extensible resource management for cluster computing

Advanced general purpose parallel systems should be able to support diverse applications with different resource requirements without compromising effectiveness and efficiency. We present a resource management model for cluster computing that allows multiple scheduling policies to co-exist dynamically. In particular, we have built Octopus, an extensible and distributed hierarchical scheduler that implements new space sharing, gang scheduling and load sharing strategies. A series of experiments performed on an IBM SP2 suggest that Octopus can effectively match application requirements to available resources, and improve the performance of a variety of parallel applications within a cluster