Using Dataflow Information to Improve Inter-Workflow Instance Concurrency

The control-flow-based design of traditional batch scheduling systems (i.e., Job A must finish before Job B is started) can constrain the concurrency in schedulers for high-performance computing (HPC) workloads. There are two main problems. First, the control-flow graph, representing the workflow, may be inherently limited in its degree of concurrency. Second, if the naming strategy of the input and output files of jobs is simplistic, there may be a filename conflict problem when multiple instances of the same workflow run concurrently.