Chunked prefill: serve prompts longer than one batch #122
Loading…
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Problem
Prefill is not chunked, so a prompt must fit in one step and
--max-batch-tokensis also the prompt ceiling (TODO atinternal/scheduler/scheduler.go:273). Batch scratch is capped at half the post-reserve memory (computeKVBudget), which makes long prompts unservable although the KV cache has plenty of room.A 104,827-token Nemotron request is refused with
request too large ... max batch size is 89801, with 2.1 GiB of KV in use and tens of GiB idle. A 144k-token prompt is out of reach even if #89 is fixed. A Granite server sized for a 70k prompt holds 18.9 GiB of scratch for one request.The Nemotron CPU reference already validates chunked prefill at the model level against HF; the missing part is scheduling and sizing.
Goal
Sequences longer than the batch budget prefill in pieces, for admission and for preemption recompute.
Acceptance
gllm plan, the status endpoint and--max-batch-tokenshelp no longer describe the batch budget as a prompt ceiling.Chunking does not reduce the total attention work in prefill; that is tracked separately.
Measurement conditions
gllm
00f89b1, RTX PRO 6000 Blackwell, 300 W power cap. Numbers are one replicate each. Data and scripts are on the build host in/homes/rcsheets/gllm/experiments/weight-space/(untracked):BASELINE.md,baseline.sh,prefill.sh,baseline.py. Compare only at the same power cap: start servers with--power-budget max=300W. GPU runs need the operator's approval.