Impactful scheduling for GPU clusters

What it claims

Ai2 serves about 150 researchers on clusters of 88 to 1,024 GPUs, with requests running at 2-3x available capacity. The old scheduler used priorities and let workloads opt out of preemption. It produced three problems:

• Squatting: users parked no-op jobs because debug launches were too slow.
• Priority inflation: eventually 100% of workloads were HIGH priority.
• On-call toil: engineers spent most of their ticket time negotiating shutdowns of non-preemptible jobs on faulty hosts.

The authors frame this as a tragedy of the commons. Their fix is to make every protected GPU-hour cost the requester's budget, so gaming the scheduler costs more than arguing for a larger budget. The algorithm is not new. It is hierarchical fair-share over a time window, in the lineage of the 2009 Hadoop Fair Scheduler, SLURM Fair Tree and YARN Fair Scheduler. What is new is the inputs: the tree mirrors the research org, and the weights are budgets set by managers.

How it works

• Budgets: managers split GPU time proportionally down the tree. A CEO or lead program manager allocates across programs, a principal investigator within a program, and a lead researcher within a project. A leaf gets a fixed share of total capacity, for example 35%, regardless of other queues.
• Fair-share ordering: workloads are sorted by the ratio of actual occupancy to allocated time over a 7-day lookback. Under-served allocations go first.
• Workload priority: still accepted, but it only reorders jobs within a team.
• Two kinds of occupancy:
• Allocated time is charged to a budget and protected during the job's minimum runtime.
• Unallocated time is free, never protected, and preemptible by any allocated request. Setting minimum runtime to 0 selects it. This keeps GPUs full when funded work isn't ready.

flowchart TD
  S[Submit: min runtime + resumable?] --> Z{min runtime = 0?}
  Z -- yes --> U[Unallocated: free, always preemptible]
  Z -- no --> Q[Queue sorted by fair-share ratio]
  Q --> R[Run protected min runtime, charged]
  R --> C{Still prioritized?}
  C -- yes --> K[Keep running, charged] --> C
  C -- no --> P[Preempt, requeue if resumable] --> Q
  K --> D[Complete]

The maximum minimum runtime is capped at 8 hours. Hosts with faults drain once their jobs reach minimum runtime, which is what allowed repairs to be automated.

Evidence

• Owed GPU hours (30 days; "owed" means allocation capped hourly at actual demand): 98% delivered overall. 13 of 15 team allocations received at least 95%, and the worst received 90%.
• Occupancy: 98% both before and after the change, with demand at 2-3x capacity in both periods. 18% of delivered time was unallocated.
• Debug jobs (small GPU count, minimum runtime of 15 minutes or less): p90 queue wait fell from about 2 hours to 30 seconds. The simulator, run on hand-built scenarios, had predicted 6 hours to 5 minutes. The authors note the baseline sample was small and high-variance.
• Largest H100 cluster: median queue wait fell from 5 minutes to 24 seconds. p90 fell from 2.8 hours to 1.8 hours.
• Repairs: repairs needing a human fell 74%.
• Simulator: an event-skipping design that takes GPU count and total runtime per job and simulates many days in a few seconds. It was used to tune the lookback window and the runtime cap, against historical traces plus constructed cases.

Limits

• Interactive sessions regressed. Researchers used to hold sessions for up to a week. Now protection ends after 8 hours, and losing a session means rebuilding its state by hand. Planned fixes are a CPU-only cluster next to on-prem storage for data-prep sessions, and restorable sessions for those CPU workloads. Neither is delivered yet.
• Possible capacity fragmentation. Wait times for the largest jobs may be growing because fewer running jobs can be interrupted at once to make room. This is unconfirmed and still under investigation.
• Adoption needed more than docs. The cluster-by-cluster rollout and leftover terminology (for example, "priority" now means something different) confused users. Live explanation sessions and dashboards showing allocation tracking and the queue-sort metric helped where documentation alone did not.
• Scope of the results. They cover one 30-day period at one organisation. The post does not describe any released code for the scheduler or the simulator.