Dynamic Scheduling of Real-Time Mixture-of-Experts Systems on Limited Resources

A Mixture-of-Experts (MoE) system generates an output in each operating cycle by combining results of multiple models (the “experts”). The contribution of any given expert to a final solution depends on a parameter called responsibility, which can vary from cycle to cycle. When resources are insufficient to run all experts, two problems arise: 1) how much utilization is to be allocated to experts and 2) how can a schedule be created based on these allocations. Problem (1) can be formulated as a succession of optimization problems, each of which calculates experts´ allocations in a cycle. Explicit mappings from responsibilities to allocation weights are needed to solve each of these problems in every cycle using a technique called “task compression (TC).” We refer to this baseline approach as TT-TC. Two other proposed heuristics TT-TC* and TT-Top reduce TC´s execution time to O for experts. To address (2), the proposed EPOC scheduler converts the heuristics´ allocations into schedules that satisfy capacity, execution, and learning constraints across cycles. Simulations demonstrate that our approaches enable real-time computation and significantly decrease the average percentage error of limited-resource outputs (i.e., 0.2%-40% and 0.3%-0.5% when scheduled with TT-TC* and TT-Top, respectively, versus 0.2%-97% when using TT-TC).