Chunked prefill: serve prompts longer than one batch #122

Open
opened 2026-11-02 22:25:37 +00:00 by rcsheets · 0 comments
Owner

Problem

Prefill is not chunked, so a prompt must fit in one step and --max-batch-tokens is also the prompt ceiling (TODO at internal/scheduler/scheduler.go:273). Batch scratch is capped at half the post-reserve memory (computeKVBudget), which makes long prompts unservable although the KV cache has plenty of room.

Model Scratch per batched token Largest batch on the Blackwell KV needed at that length
Nemotron 3 Nano about 396 KB (33.1 GiB / 89,801) 89,801 tokens about 12 KB per token
Granite 4.2 8b about 286 KB (18.9 GiB / 71,000) not reached at 71,000 about 320 KB per token

A 104,827-token Nemotron request is refused with request too large ... max batch size is 89801, with 2.1 GiB of KV in use and tens of GiB idle. A 144k-token prompt is out of reach even if #89 is fixed. A Granite server sized for a 70k prompt holds 18.9 GiB of scratch for one request.

The Nemotron CPU reference already validates chunked prefill at the model level against HF; the missing part is scheduling and sizing.

Goal

Sequences longer than the batch budget prefill in pieces, for admission and for preemption recompute.

Acceptance

  • A 144,000-token prompt serves on Nemotron with the default batch budget, and scratch stays bounded by that budget.
  • Chunked and whole prefill produce matching logits within the existing golden tolerances, for Nemotron (with recurrent state) and an attention-only model, on CPU and CUDA.
  • Other running sequences keep decoding between chunks.
  • gllm plan, the status endpoint and --max-batch-tokens help no longer describe the batch budget as a prompt ceiling.

Chunking does not reduce the total attention work in prefill; that is tracked separately.

Measurement conditions

gllm 00f89b1, RTX PRO 6000 Blackwell, 300 W power cap. Numbers are one replicate each. Data and scripts are on the build host in /homes/rcsheets/gllm/experiments/weight-space/ (untracked): BASELINE.md, baseline.sh, prefill.sh, baseline.py. Compare only at the same power cap: start servers with --power-budget max=300W. GPU runs need the operator's approval.

## Problem Prefill is not chunked, so a prompt must fit in one step and `--max-batch-tokens` is also the prompt ceiling (TODO at `internal/scheduler/scheduler.go:273`). Batch scratch is capped at half the post-reserve memory (`computeKVBudget`), which makes long prompts unservable although the KV cache has plenty of room. | Model | Scratch per batched token | Largest batch on the Blackwell | KV needed at that length | |---|---|---|---| | Nemotron 3 Nano | about 396 KB (33.1 GiB / 89,801) | 89,801 tokens | about 12 KB per token | | Granite 4.2 8b | about 286 KB (18.9 GiB / 71,000) | not reached at 71,000 | about 320 KB per token | A 104,827-token Nemotron request is refused with `request too large ... max batch size is 89801`, with 2.1 GiB of KV in use and tens of GiB idle. A 144k-token prompt is out of reach even if #89 is fixed. A Granite server sized for a 70k prompt holds 18.9 GiB of scratch for one request. The Nemotron CPU reference already validates chunked prefill at the model level against HF; the missing part is scheduling and sizing. ## Goal Sequences longer than the batch budget prefill in pieces, for admission and for preemption recompute. ## Acceptance - A 144,000-token prompt serves on Nemotron with the default batch budget, and scratch stays bounded by that budget. - Chunked and whole prefill produce matching logits within the existing golden tolerances, for Nemotron (with recurrent state) and an attention-only model, on CPU and CUDA. - Other running sequences keep decoding between chunks. - `gllm plan`, the status endpoint and `--max-batch-tokens` help no longer describe the batch budget as a prompt ceiling. Chunking does not reduce the total attention work in prefill; that is tracked separately. ## Measurement conditions gllm `00f89b1`, RTX PRO 6000 Blackwell, 300 W power cap. Numbers are one replicate each. Data and scripts are on the build host in `/homes/rcsheets/gllm/experiments/weight-space/` (untracked): `BASELINE.md`, `baseline.sh`, `prefill.sh`, `baseline.py`. Compare only at the same power cap: start servers with `--power-budget max=300W`. GPU runs need the operator's approval.
Sign in to join this conversation.
No labels
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
brooktrails/gllm#122
No description provided.