Optimization Patterns for the Decentralized Orchestration of Parameter-Sweep Workflows

A large and diverse group of computational scientific research efforts deal with parameterized studies, in which same or similar computational tools are applied on different sets of data. Such uniform and well-defined analysis efforts can be encapsulated as parameter-sweep workflows. Due to computation and data intensive nature, resources that span across multiple domains may be needed for timely and efficient execution of this type of workflows. In our previous studies, we have designed and developed techniques to orchestrate the execution of large-scale workflows in a decentralized and adaptive manner. Through the usage of generic workflow patterns, centralized orchestration of workflows are transformed into decentralized and adaptive orchestration without modifying the business logic of the workflow. In this study, we propose some additional optimization patterns specific to characteristics and requirements of parameter-sweep workflows. By exploiting the general characteristics of parameter-sweep workflows, we provide ways to reduce control and data overheads associated with the decentralized orchestration. We also discuss some implementation issues that arise from the adoption of these optimization patterns.