Decode step time grows too fast with context: 5.2 ms per 1k tokens on Granite #124

Open
opened 2026-11-02 22:25:38 +00:00 by rcsheets · 1 comment
Owner

Problem

Decode step time grows linearly with sequence length, which makes long replies after long prompts extremely expensive.

Model ms per token, concurrency 1
Nemotron 3 Nano 6.4 + 0.78 per 1k tokens of context
Granite 4.2 8b 14.6 + 5.2 per 1k tokens of context

Extrapolated (not measured), a 32,768-token reply takes about 4.2 h after a 70k Granite prompt, 5.5 h after 96k, and 55 min after a 105k Nemotron prompt. Reasoning replies do run this long: in 16 short-prompt requests, 9 were still thinking when the time limit stopped them, at about 23k tokens on Nemotron and 8.6k on Granite.

Anomaly to explain

On Granite, concurrency 8 costs nearly the same per step as concurrency 1 at the same length: 75.0 vs 74.0 ms at about 8.5k tokens, with mean power 300 W vs 215 W. Eight times the attention work takes the same time, so single-sequence decode appears to be leaving the GPU underused. Nemotron does not show this at short context (20.4 vs 10.4 ms).

Likely cause (not yet confirmed)

Linear growth is expected from reading the whole KV span, but its cost looks high. Candidates: paged KV block indirection in paged_attention_flash_kernel, its warp parallelism for a single long span, and host work per step that scales with span length.

First step

Profile decode with nsys launch at about 8k and 30k tokens of context, at concurrency 1 and 8. The 300-token runs in prefill.sh reach these contexts quickly.

Acceptance

  • The c1 vs c8 result is explained, with profiles attached.
  • Granite 4.2 8b's growth falls from 5.2 to at most 1.5 ms per 1k tokens at concurrency 1, at 300 W, with greedy output within golden tolerances.
  • No regression in the short-context decode benchmark.

This touches the same kernels as the prefill attention work, so coordinate changes to paged_attention_flash_kernel.

Measurement conditions

gllm 00f89b1, RTX PRO 6000 Blackwell, 300 W power cap. Numbers are one replicate each. Data and scripts are on the build host in /homes/rcsheets/gllm/experiments/weight-space/ (untracked): BASELINE.md, baseline.sh, prefill.sh, baseline.py. Compare only at the same power cap: start servers with --power-budget max=300W. GPU runs need the operator's approval.

## Problem Decode step time grows linearly with sequence length, which makes long replies after long prompts extremely expensive. | Model | ms per token, concurrency 1 | |---|---| | Nemotron 3 Nano | 6.4 + 0.78 per 1k tokens of context | | Granite 4.2 8b | 14.6 + 5.2 per 1k tokens of context | Extrapolated (not measured), a 32,768-token reply takes about 4.2 h after a 70k Granite prompt, 5.5 h after 96k, and 55 min after a 105k Nemotron prompt. Reasoning replies do run this long: in 16 short-prompt requests, 9 were still thinking when the time limit stopped them, at about 23k tokens on Nemotron and 8.6k on Granite. ## Anomaly to explain On Granite, concurrency 8 costs nearly the same per step as concurrency 1 at the same length: 75.0 vs 74.0 ms at about 8.5k tokens, with mean power 300 W vs 215 W. Eight times the attention work takes the same time, so single-sequence decode appears to be leaving the GPU underused. Nemotron does not show this at short context (20.4 vs 10.4 ms). ## Likely cause (not yet confirmed) Linear growth is expected from reading the whole KV span, but its cost looks high. Candidates: paged KV block indirection in `paged_attention_flash_kernel`, its warp parallelism for a single long span, and host work per step that scales with span length. ## First step Profile decode with `nsys launch` at about 8k and 30k tokens of context, at concurrency 1 and 8. The 300-token runs in `prefill.sh` reach these contexts quickly. ## Acceptance - The c1 vs c8 result is explained, with profiles attached. - Granite 4.2 8b's growth falls from 5.2 to at most 1.5 ms per 1k tokens at concurrency 1, at 300 W, with greedy output within golden tolerances. - No regression in the short-context decode benchmark. This touches the same kernels as the prefill attention work, so coordinate changes to `paged_attention_flash_kernel`. ## Measurement conditions gllm `00f89b1`, RTX PRO 6000 Blackwell, 300 W power cap. Numbers are one replicate each. Data and scripts are on the build host in `/homes/rcsheets/gllm/experiments/weight-space/` (untracked): `BASELINE.md`, `baseline.sh`, `prefill.sh`, `baseline.py`. Compare only at the same power cap: start servers with `--power-budget max=300W`. GPU runs need the operator's approval.
Author
Owner

Measurements at a 350 W cap, with nsys kernel profiles and a vLLM 0.33.0 reference on the same prompts and the same card. Step times are from 290 tokens after the first ten of a 300-token reply, concurrency 1.

Model Context tokens gllm ms/token vLLM ms/token Ratio
Nemotron 6,378 11.8 3.2 3.7x
Nemotron 30,802 30.8 3.2 9.6x
Nemotron 87,946 75.5 3.3 23x
Granite 7,218 52.3 12.9 4.1x
Granite 15,320 94.6 13.8 6.9x
Granite 31,513 not run 15.5

So decode cost does not have to grow like this: vLLM's Nemotron step is nearly flat from 6k to 88k tokens. Raising the cap from 300 W to 350 W made no difference to c1 decode (Granite at 15k: 94.5 vs 94.6 ms), so this is not power-limited.

Kernel profile. paged_attention_flash_kernel share of kernel time, and time per launch:

Case Share ms/launch
Granite c1, 15k 86.5% 2.03
Granite c1, 7k 76.2% 1.007
Granite c8, 7k 76.4% 1.056
Nemotron c1, 31k 84.9% 4.14
Nemotron c1, 6k 60.2% 1.11
Nemotron c8, 6k 30.4% 0.97

This explains why c1 and c8 cost the same. Eight sequences take about the same time per launch as one, measured directly above (1.007 vs 1.056 ms on Granite). The launch in kernels.cu is one launch per layer with dim3 grid(numTokens, qHeads) and a block of ATTN_WARPS*ATTN_LANES (8 warps by 32 lanes). Each block's 8 warps cross the whole KV span serially, so launch time follows the longest span, not the total work. One token with 32 query heads launches only 32 blocks, which leaves most of the GPU idle. Eight tokens launch 256 blocks in about the same time. This is read from the code and consistent with the timings; it has not been confirmed by an experiment.

Suggested direction. Split a long span across many more blocks (tile each span, then combine the online-softmax partial results in order). Effective rates from these runs are about 7.3e6 (query, position) pairs/s at c1 and 5.5e7 at c8, against 1.05e8 in prefill, so the room at c1 is large if tiling works. vLLM's log shows it uses FlashInfer's XQA decode backend (head_group_ratio_16, E4M3 KV cache), which is the reference to compare against.

Why this matters for long replies. Twelve short-prompt reasoning tasks ran on Nemotron at concurrency 8 with a 32,768-token limit. Six finished (651 to 25,424 tokens) and six reached the limit with no answer yet, taking about 18.5 minutes. Per-sequence step time rose from about 21.6 ms early to 44 ms by the end of those runs. Long decode is the normal case for this workload, not an edge case.

Measurements at a 350 W cap, with nsys kernel profiles and a vLLM 0.33.0 reference on the same prompts and the same card. Step times are from 290 tokens after the first ten of a 300-token reply, concurrency 1. | Model | Context tokens | gllm ms/token | vLLM ms/token | Ratio | |---|---|---|---|---| | Nemotron | 6,378 | 11.8 | 3.2 | 3.7x | | Nemotron | 30,802 | 30.8 | 3.2 | 9.6x | | Nemotron | 87,946 | 75.5 | 3.3 | 23x | | Granite | 7,218 | 52.3 | 12.9 | 4.1x | | Granite | 15,320 | 94.6 | 13.8 | 6.9x | | Granite | 31,513 | not run | 15.5 | | So decode cost does not have to grow like this: vLLM's Nemotron step is nearly flat from 6k to 88k tokens. Raising the cap from 300 W to 350 W made no difference to c1 decode (Granite at 15k: 94.5 vs 94.6 ms), so this is not power-limited. **Kernel profile.** `paged_attention_flash_kernel` share of kernel time, and time per launch: | Case | Share | ms/launch | |---|---|---| | Granite c1, 15k | 86.5% | 2.03 | | Granite c1, 7k | 76.2% | 1.007 | | Granite c8, 7k | 76.4% | 1.056 | | Nemotron c1, 31k | 84.9% | 4.14 | | Nemotron c1, 6k | 60.2% | 1.11 | | Nemotron c8, 6k | 30.4% | 0.97 | **This explains why c1 and c8 cost the same.** Eight sequences take about the same time per launch as one, measured directly above (1.007 vs 1.056 ms on Granite). The launch in `kernels.cu` is one launch per layer with `dim3 grid(numTokens, qHeads)` and a block of `ATTN_WARPS*ATTN_LANES` (8 warps by 32 lanes). Each block's 8 warps cross the whole KV span serially, so launch time follows the longest span, not the total work. One token with 32 query heads launches only 32 blocks, which leaves most of the GPU idle. Eight tokens launch 256 blocks in about the same time. This is read from the code and consistent with the timings; it has not been confirmed by an experiment. **Suggested direction.** Split a long span across many more blocks (tile each span, then combine the online-softmax partial results in order). Effective rates from these runs are about 7.3e6 (query, position) pairs/s at c1 and 5.5e7 at c8, against 1.05e8 in prefill, so the room at c1 is large if tiling works. vLLM's log shows it uses FlashInfer's XQA decode backend (`head_group_ratio_16`, E4M3 KV cache), which is the reference to compare against. **Why this matters for long replies.** Twelve short-prompt reasoning tasks ran on Nemotron at concurrency 8 with a 32,768-token limit. Six finished (651 to 25,424 tokens) and six reached the limit with no answer yet, taking about 18.5 minutes. Per-sequence step time rose from about 21.6 ms early to 44 ms by the end of those runs. Long decode is the normal case for this workload, not an edge case.
Sign in to join this conversation.
No labels
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
brooktrails/gllm#124
No description provided.