A Passive Solution to the Memory Resource Discovery Problem in Computational Clusters

Resource discovery is an important problem in distributed computing, because the throughput of the system is directly linked to its ability to quickly locate available resources. Current solutions are undesirable for discovering resources in large computational clusters because they are intrusive, chatty (i.e., have per-node overhead), or maintenance-intensive. In this paper, we present a novel method that offers the ability to non-intrusively identify resources that have available memory; this is critical for memory-intensive cluster applications such as weather forecasting and computational chemistry. The prime benefits are fourfold: (1) low message complexity, (2) scalability, (3) load balancing, and (4) low maintainability. We demonstrate the feasibility of our method with experiments using a 50-node test-bed (DETERlab). Our technique allows us to establish a correlation between memory load and the timely response of network traffic from a node. Results show that our method can accurately (92%-100%) identify nodes with available memory through analysis of existing network traffic, including network traffic that has passed through a switch (non-congested).