Constraint-Aware GPU Allocator Improves Utilization by 33 Points Over FIFO Scheduling

Improved GPU Utilization and Output

A constraint-aware GPU allocator was benchmarked against a traditional FIFO (First-In, First-Out) scheduler across seven scenarios. The new allocator achieved a significant increase in GPU utilization, by as much as 33 percentage points. Additionally, the priority-weighted output saw an increase of up to 105% in every benchmark scenario.

Impact of Allocation Order

These gains were realized on identical hardware running identical workloads, indicating that the improvement stems solely from the order in which allocation decisions are made. The study emphasizes that simply 'keeping GPUs busy' is not sufficient; the specific assignment of jobs to GPUs, at what time, and with what priority, is crucial for efficiency.

Addressing Workload Heterogeneity

The core problem addressed by the allocator is the competition between different workload types for the same hardware. Workloads like training, batch inference, and quantization require contiguous blocks of GPUs for extended periods, while real-time inference is elastic and demand-driven. Managing these incompatible allocation shapes, along with varying job durations and GPU requirements within a single type, is key to optimizing cluster performance.

Limitations of FIFO Scheduling

The FIFO-based scheduler used for comparison typically serves real-time inference from a fixed reservation and places other jobs in arrival order without considering priority. While reasonable under low contention, this policy becomes inefficient and costly in terms of capacity when the cluster experiences high demand, as the ordering cost becomes visible and impacts overall utilization.