Kueue¶
FLAME uses Kueue — a Kubernetes-native job queueing system — to decide which GPU workloads run, when, and on which hardware. You rarely interact with Kueue directly: it works behind the scenes when you submit a Job or TrainJob, enforcing your group's allocation and the cluster's fair-sharing rules.
This page explains the handful of Kueue concepts that show up elsewhere in these docs.
Resource flavors¶
A resource flavor (Kueue's ResourceFlavor)
is a named variety of a resource that distinguishes otherwise-identical
resource types by which machines they live on.
A plain Kubernetes resource like nvidia.com/gpu just means "a GPU" — it says
nothing about what kind. A flavor adds that distinction. On FLAME, the gh200
flavor means "an nvidia.com/gpu that is specifically a GH200, on one of the
nodes that has GH200s in it."
Flavors are matched to nodes by node labels. FLAME labels each GPU node with
the type of GPU it contains, and each flavor selects the nodes whose labels
match. So when a workload asks for the gh200 flavor, Kueue knows it can only be
placed on the physical GH200 machines — no other node qualifies.
This is why allocations and workloads name a flavor (gh200) rather than just a
raw resource (nvidia.com/gpu): the flavor is what pins your request to the
right hardware. See Allocations → What an allocation
is for how this fits together.
Today there is only one flavor
FLAME currently defines a single flavor, gh200. As the cluster grows — more
GPU models, or MIG slices that carve
one GPU into smaller pieces — additional flavors will appear, each tagging a
different slice of the hardware.
Cohorts and fair sharing¶
All of FLAME's GPU capacity lives in a single shared cohort (Kueue's term for a pool of capacity that groups can borrow across). Each research group has a queue within this cohort, and Kueue continuously balances the pool: idle guaranteed capacity can be borrowed, guaranteed capacity is reclaimed on demand, and the borrowed pool is split fairly between competing borrowers.
The allocation data model is simple — the interesting part is what happens when several groups submit work at once. Let's walk through it.
Imagine the cluster has 8 GPUs and three groups hold allocations:
| Group | Allocation | Guaranteed |
|---|---|---|
| A | Standard | 4 GPUs |
| B | Standard | 2 GPUs |
| C | Borrowing-only | none |
Guaranteed capacity adds up to 6 of the 8 GPUs; the other 2 are unallocated and free for anyone to borrow.
1. Borrowing idle capacity¶
Group A submits nothing for now, Group B runs its 2 guaranteed GPUs, and Group C submits 4 jobs. Since A's 4 guaranteed GPUs are sitting idle, C is allowed to borrow them (along with the unallocated capacity):
Guarantees: A = 4 B = 2 C = borrowing-only
Submitted: A: — B: 2 C: 4
┌───┬───┬───┬───┬───┬───┬───┬───┐
│ C │ C │ C │ C │ B │ B │ · │ · │ · = idle
└───┴───┴───┴───┴───┴───┴───┴───┘
C borrowing 4 B's 2 2 free
The cluster stays busy instead of leaving A's reserved GPUs idle. Everything is fine — until A wants its GPUs.
2. Reclaim and preemption¶
Group A now submits 4 jobs. Its allocation guarantees 4 GPUs, so it must get them immediately. Only 2 GPUs are free, so the system preempts 2 of Group C's borrowed jobs to make room. (Group B is untouched — it's within its own guarantee, so it's never a preemption target.)
A submits 4 jobs → it reclaims its 4 guaranteed GPUs.
Suspended & re-queued: C: ▣ ▣
┌───┬───┬───┬───┬───┬───┬───┬───┐
│ A │ A │ A │ A │ B │ B │ C │ C │
└───┴───┴───┴───┴───┴───┴───┴───┘
A's guaranteed 4 B's 2 C: 2 left
Group A gets all 4 GPUs instantly; Group C drops from 4 borrowed GPUs to 2. The two preempted jobs are suspended and put back in the queue — not failed.
3. Automatic resume¶
When Group A's jobs finish, those GPUs return to the pool, and Group C's suspended jobs resume automatically from their last checkpoint — no resubmission needed:
A's jobs finish → C's suspended jobs resume.
┌───┬───┬───┬───┬───┬───┬───┬───┐
│ · │ · │ C │ C │ B │ B │ C │ C │
└───┴───┴───┴───┴───┴───┴───┴───┘
2 free B's 2 C borrowing 4 again
Fair sharing between borrowers¶
One more wrinkle: if two groups want to borrow more than is free at the same time, the system doesn't let whoever asked first grab everything. It splits the idle pool between them fairly and continuously — if one borrower is using a large share and another submits new work, some of the first borrower's jobs are preempted to even things out. Guaranteed capacity is never touched by this; fair sharing only ever rebalances the borrowed pool.
Because borrowed capacity can be reclaimed at any time, any workload that borrows must be built to survive eviction. See Designing GPU workloads for preemption for how.