Long prefill is quadratic and slow: 70k tokens takes over 10 minutes on Granite at 300 W #123
Loading…
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Problem
Prefill time grows with about the square of prompt length, and is very slow at long context.
A 70,307-token Granite prompt had not finished at 600 s. Extrapolating the fit (not measured): Granite 70k about 19 min, Granite 96k about 36 min, Nemotron 105k about 6.5 min.
The core clock sits near 1,040 MHz at the full 300 W during prefill, against 2,730-2,835 MHz during decode, so this workload is power-limited rather than clock-limited.
Likely cause (not yet confirmed)
Exact attention is inherently quadratic, so the exponent may not change; the constant is the target. An earlier profile of a 4,314-token Nemotron prefill put about 0.47 s of 1.40 s in attention, running through the decode flash kernel (
paged_attention_flash_kernel), but no profile exists at long context.First step
Profile a Granite prefill at about 15k tokens and a Nemotron prefill at about 30k with
nsys launchplus start/stop around the request.nsys profileloses CUPTI data when the Go process exits.Acceptance
Measurement conditions
gllm
00f89b1, RTX PRO 6000 Blackwell, 300 W power cap. Numbers are one replicate each. Data and scripts are on the build host in/homes/rcsheets/gllm/experiments/weight-space/(untracked):BASELINE.md,baseline.sh,prefill.sh,baseline.py. Compare only at the same power cap: start servers with--power-budget max=300W. GPU runs need the operator's approval.Measurements at a 350 W cap, with nsys kernel profiles and a vLLM 0.33.0 reference on the same prompts and the same card.
Where the time goes.
paged_attention_flash_kernelis 97.0% of kernel time in a 15k-token Granite prefill (1.10 s per layer, 40 layers) and 92.7% in a 31k-token Nemotron prefill (4.53 s per attention layer, 6 layers).Measured times, seconds to first token
Nemotron's growth from 30.8k to 87.9k has an exponent of 1.97, so prefill is still quadratic up to the longest prompt accepted today. Raising the cap from 300 W to 350 W (17%) made prefill 14-16% faster.
vLLM long-prompt numbers are not cold. Only the first prompt per model is a cold comparison. The prompts are prefixes of one another, vLLM's prefix cache was on, and its log confirms hits (31.9% on Granite), so each later request prefilled only its new tokens:
Summing the increments estimates a cold time of about 3.7 s for Nemotron 87,946 tokens against 236 s in gllm (about 65x), and 3.0 s for Granite 15,320 against 47.5 s (about 16x). These sums are estimates, not cold measurements. vLLM ran with chunked prefill at
max_num_batched_tokens=8192, and its per-request time stays nearly linear with new tokens across these lengths, so the quadratic growth is not inherent to the workload. Chunking alone does not explain it, since each chunk still attends to all earlier tokens. vLLM's log shows FlashInfer prefill with an E4M3 KV cache and bfloat16 queries, which is a lead to examine.Power during these runs was at most 358 W at 1 Hz for vLLM, and at most 364.6 W for gllm against the 350 W cap.