The organic grid: self-organizing computation on a peer-to-peer network

Desktop grids have recently been used to perform some of the largest computations in the world and have the potential to grow by several more orders of magnitude. However, current approaches to utilizing desktop resources require either centralized servers or extensive knowledge of the underlying system, limiting their scalability. We propose a biologically inspired and fully-decentralized approach to the organization of computation that is based on the autonomous scheduling of strongly mobile agents on a peer-to-peer network. In a radical departure from current models, we envision large-scale desktop grids in which agents autonomously organize themselves so as to maximize resource utilization. By encapsulating computation and behavior into agents, the organization of the computation can be customized for different classes of applications. At the same time, the design of the underlying infrastructure is greatly simplified, resulting in a system that naturally lends itself to a true peer-to-peer implementation where each node can be at the same time provider and user of the computing utility infrastructure. We demonstrate this concept with a reduced-scale proof-of-concept implementation that executes a data-intensive independent-task application on a set of heterogeneous, geographically distributed machines. We present a detailed exploration of the design space of our system and a performance evaluation of our implementation using metrics appropriate for assessing self-organizing desktop grids.