Intelligent job selection for distributed scheduling

A key issue in distributed scheduling is selecting appropriate jobs to transfer. A job selection policy that considers the diversity of job behaviors is proposed. A mechanism used in artificial neural networks, called weight climbing, is employed. Using this mechanism, a distributed scheduler can learn the behavior of a job from its past executions and make a correct prediction about whether transferring the job is worthwhile. A scheduler using the proposed job selection policy has been implemented and experimental results show that it is able to learn job behaviors fast, make decisions accurately and adjust itself promptly when system configuration or program behaviors are changed. In addition, the selection policy introduces only negligible time and space overhead