CrossTree: A new HTC architecture with high reliability and scalability

HTC (high throughput computing) is a environment that can provide large amounts of processing capacity over long periods of time. To HTC, users are more concerned about how many jobs can be completed in a long period, but not how fast can a single job be finished. Condor, an implementation of HTC, is constructed by commodity CPUs and memory. As long as the Condor nodes are controlled by the Central Management Node, its reliability and scalability had been restricted. Based on the concept of DHT (distributed hash table), this paper presents a new distributed HTC architecture, named CrossTree, which has no central parts, and its metadata is distributed across all nodes in the system. Theoretical analysis and the simulation results proved CrossTree to be an efficient architecture with high scalability and reliability.