Decode step time grows too fast with context: 5.2 ms per 1k tokens on Granite #124
Loading…
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Problem
Decode step time grows linearly with sequence length, which makes long replies after long prompts extremely expensive.
Extrapolated (not measured), a 32,768-token reply takes about 4.2 h after a 70k Granite prompt, 5.5 h after 96k, and 55 min after a 105k Nemotron prompt. Reasoning replies do run this long: in 16 short-prompt requests, 9 were still thinking when the time limit stopped them, at about 23k tokens on Nemotron and 8.6k on Granite.
Anomaly to explain
On Granite, concurrency 8 costs nearly the same per step as concurrency 1 at the same length: 75.0 vs 74.0 ms at about 8.5k tokens, with mean power 300 W vs 215 W. Eight times the attention work takes the same time, so single-sequence decode appears to be leaving the GPU underused. Nemotron does not show this at short context (20.4 vs 10.4 ms).
Likely cause (not yet confirmed)
Linear growth is expected from reading the whole KV span, but its cost looks high. Candidates: paged KV block indirection in
paged_attention_flash_kernel, its warp parallelism for a single long span, and host work per step that scales with span length.First step
Profile decode with
nsys launchat about 8k and 30k tokens of context, at concurrency 1 and 8. The 300-token runs inprefill.shreach these contexts quickly.Acceptance
This touches the same kernels as the prefill attention work, so coordinate changes to
paged_attention_flash_kernel.Measurement conditions
gllm
00f89b1, RTX PRO 6000 Blackwell, 300 W power cap. Numbers are one replicate each. Data and scripts are on the build host in/homes/rcsheets/gllm/experiments/weight-space/(untracked):BASELINE.md,baseline.sh,prefill.sh,baseline.py. Compare only at the same power cap: start servers with--power-budget max=300W. GPU runs need the operator's approval.Measurements at a 350 W cap, with nsys kernel profiles and a vLLM 0.33.0 reference on the same prompts and the same card. Step times are from 290 tokens after the first ten of a 300-token reply, concurrency 1.
So decode cost does not have to grow like this: vLLM's Nemotron step is nearly flat from 6k to 88k tokens. Raising the cap from 300 W to 350 W made no difference to c1 decode (Granite at 15k: 94.5 vs 94.6 ms), so this is not power-limited.
Kernel profile.
paged_attention_flash_kernelshare of kernel time, and time per launch:This explains why c1 and c8 cost the same. Eight sequences take about the same time per launch as one, measured directly above (1.007 vs 1.056 ms on Granite). The launch in
kernels.cuis one launch per layer withdim3 grid(numTokens, qHeads)and a block ofATTN_WARPS*ATTN_LANES(8 warps by 32 lanes). Each block's 8 warps cross the whole KV span serially, so launch time follows the longest span, not the total work. One token with 32 query heads launches only 32 blocks, which leaves most of the GPU idle. Eight tokens launch 256 blocks in about the same time. This is read from the code and consistent with the timings; it has not been confirmed by an experiment.Suggested direction. Split a long span across many more blocks (tile each span, then combine the online-softmax partial results in order). Effective rates from these runs are about 7.3e6 (query, position) pairs/s at c1 and 5.5e7 at c8, against 1.05e8 in prefill, so the room at c1 is large if tiling works. vLLM's log shows it uses FlashInfer's XQA decode backend (
head_group_ratio_16, E4M3 KV cache), which is the reference to compare against.Why this matters for long replies. Twelve short-prompt reasoning tasks ran on Nemotron at concurrency 8 with a 32,768-token limit. Six finished (651 to 25,424 tokens) and six reached the limit with no answer yet, taking about 18.5 minutes. Per-sequence step time rose from about 21.6 ms early to 44 ms by the end of those runs. Long decode is the normal case for this workload, not an edge case.