Enhancing cluster application performance via smarter scheduling and stronger SOAP

Current systems for managing workload on clusters of workstations, particularly those available for Linux-based (Beowulf) clusters, are typically based on traditional process-based, coarse-grained parallel and distributed programming. The DESPOT project is building a sophisticated resource-monitoring and scheduling system for computational, storage and network resources. In this paper we present our enhanced scheduling algorithm within DESPOT, our architecture for low-overhead, fine-grained resource-monitoring tools for per-process network and other resource usage. This algorithm takes into account network, CPU, and memory demands to go beyond simple load balancing for better performance. We also present our architecture for enhancing inter-process communication using the SOAP protocol through the application of multi-level caching. We present experimental results on a small Beowulf cluster showing scheduling performance gains of up to 40%, using the multi-facetted DESPOT algorithm over the MOSIX load-balancing implementation, and SOAP performance gains of up to 600%.