Effective utilization and reconfiguration of distributed hardware resources using job management systems

Reconfigurable hardware resources are very expensive, and yet can be underutilized. This paper describes a middleware capable of discovering underutilized computing nodes with FPGA-based accelerator boards in a networked environment. Using an extended job management system (JMS), this middleware permits sharing reconfigurable resources at least among the members of the same organization. Traditional resources, such as CPU time of loosely coupled workstations can be shared using a variety of existing job management systems (JMS). We analyzed four of these systems, LSF, Sun Grid Engine/CODINE, PBS Pro, and Condor from the point of view of their functional characteristics and ease of extension to support reconfigurable hardware. LSF was shown to efficiently address the majority of identified requirements. The general architecture of the extended system was developed, and the exact techniques of extending LSF, CODINE, and PBS Pro to manage FPGA-based accelerator boards were identified. The system architecture was verified experimentally for the specific case of LSF and three types of FPGA accelerator boards. The utilization of FPGA boards was demonstrated to reach up to 86% in our experimental setting consisting of Linux and Windows NT workstations.