GQA decode attention: c1 short-context decode 19% slower on Nemotron since v0.54.2 #131

Open
opened 2026-11-03 09:16:12 +00:00 by rcsheets · 0 comments
Owner

Summary

v0.54.2 (the GQA-aware decode attention from #129) decodes Nemotron 3 Nano about 19% slower than v0.54.1 for a single request at short context: 7.86 ms per token against 6.63 at a 6,378-token prompt. Longer contexts and batched decode got faster, so the loss is confined to small launches at short span. Single-user chats at a few thousand tokens are a common case, so this is worth recovering.

Measurements

RTX PRO 6000 Blackwell (GPU-bc3460ab-6ad9-c6ef-c955-6922515f2b7f), power capped at 350 W, no other process on the card. Both binaries were built with make cuda from the release tags. Model: NVIDIA-Nemotron-3-Nano-30B-A3B-NVFP4 (32 query heads, 2 KV heads, GQA group 16).

Short-context repeat: one request at a time with a 6,378-token prompt, 300 completion tokens per request, and ms/token measured from the 11th token to the last. Ten requests per version, with the two servers alternated over two rounds of five:

Version n ms/token, mean ± sd Range
v0.54.1 10 6.63 ± 0.01 6.61–6.65
v0.54.2 10 7.86 ± 0.02 7.84–7.89

The two rounds agree to within 0.03 ms, so this is not drift.

Same-session sweep with one run per point. Decode is measured after every request in the batch has received its first token.

Case (Nemotron) v0.54.1 v0.54.2
c1, 6k context 6.6 7.9
c1, 31k context 7.9 7.9
c1, 88k context 12.2 8.5
c8, 6k context 20.3 18.9
c16, 31k context 66.4 37.8

Granite 4.2 8b (group 4, 8 KV heads) shows no regression at c1 7k (17.6 vs 16.9).

Likely cause (not yet profiled)

The tile count comes from attn_tiles(numTokens, kvHeads, maxSpan), which takes the smaller of two values:

  • ATTN_TILE_BLOCKS / (numTokens * kvHeads) blocks, and
  • maxSpan / ATTN_TILE_MIN_SPAN (512).

For Nemotron at c1 with a 6,378-token span:

  • v0.54.1 tiled per query head: min(2048/32, 12) = 12 tiles × 32 heads = 384 blocks.
  • v0.54.2 tiles per KV head: min(1024, 12) = 12 tiles × 2 KV heads = 24 blocks.

That is 24 blocks on a GPU with roughly 188 SMs. The minimum-span limit binds before the block target does. Each GQA block now does group times the work per position, yet the 512-position minimum is unchanged.

At 31k the same limit gives 60 × 2 = 120 blocks, which is where the two versions break even. At 88k it gives 342 blocks, and v0.54.2 is ahead there.

Possible fixes

  • Scale the useful-tile bound by the work per position, for example maxSpan * group / ATTN_TILE_MIN_SPAN, or lower the minimum span on the GQA path.
  • Alternatively, keep the per-query-head kernel when numTokens * kvHeads * tiles would leave most SMs idle.

The first fix is a one-line change. To test it, sweep gllm_set_attn_tile_min_span at c1 for 6k, 31k and 88k, and check that c8 and c16 do not regress.

Acceptance

At 350 W on Nemotron, c1 decode at 6k context is at or below v0.54.1's 6.63 ms per token. The v0.54.2 gains at 31k and 88k, and under batched decode, are kept.

## Summary v0.54.2 (the GQA-aware decode attention from #129) decodes Nemotron 3 Nano about 19% slower than v0.54.1 for a single request at short context: 7.86 ms per token against 6.63 at a 6,378-token prompt. Longer contexts and batched decode got faster, so the loss is confined to small launches at short span. Single-user chats at a few thousand tokens are a common case, so this is worth recovering. ## Measurements RTX PRO 6000 Blackwell (`GPU-bc3460ab-6ad9-c6ef-c955-6922515f2b7f`), power capped at 350 W, no other process on the card. Both binaries were built with `make cuda` from the release tags. Model: NVIDIA-Nemotron-3-Nano-30B-A3B-NVFP4 (32 query heads, 2 KV heads, GQA group 16). Short-context repeat: one request at a time with a 6,378-token prompt, 300 completion tokens per request, and ms/token measured from the 11th token to the last. Ten requests per version, with the two servers alternated over two rounds of five: | Version | n | ms/token, mean ± sd | Range | |---|---|---|---| | v0.54.1 | 10 | 6.63 ± 0.01 | 6.61–6.65 | | v0.54.2 | 10 | 7.86 ± 0.02 | 7.84–7.89 | The two rounds agree to within 0.03 ms, so this is not drift. Same-session sweep with one run per point. Decode is measured after every request in the batch has received its first token. | Case (Nemotron) | v0.54.1 | v0.54.2 | |---|---|---| | c1, 6k context | 6.6 | 7.9 | | c1, 31k context | 7.9 | 7.9 | | c1, 88k context | 12.2 | 8.5 | | c8, 6k context | 20.3 | 18.9 | | c16, 31k context | 66.4 | 37.8 | Granite 4.2 8b (group 4, 8 KV heads) shows no regression at c1 7k (17.6 vs 16.9). ## Likely cause (not yet profiled) The tile count comes from `attn_tiles(numTokens, kvHeads, maxSpan)`, which takes the smaller of two values: - `ATTN_TILE_BLOCKS / (numTokens * kvHeads)` blocks, and - `maxSpan / ATTN_TILE_MIN_SPAN` (512). For Nemotron at c1 with a 6,378-token span: - **v0.54.1** tiled per query head: min(2048/32, 12) = 12 tiles × 32 heads = **384 blocks**. - **v0.54.2** tiles per KV head: min(1024, 12) = 12 tiles × 2 KV heads = **24 blocks**. That is 24 blocks on a GPU with roughly 188 SMs. The minimum-span limit binds before the block target does. Each GQA block now does `group` times the work per position, yet the 512-position minimum is unchanged. At 31k the same limit gives 60 × 2 = 120 blocks, which is where the two versions break even. At 88k it gives 342 blocks, and v0.54.2 is ahead there. ## Possible fixes - Scale the useful-tile bound by the work per position, for example `maxSpan * group / ATTN_TILE_MIN_SPAN`, or lower the minimum span on the GQA path. - Alternatively, keep the per-query-head kernel when `numTokens * kvHeads * tiles` would leave most SMs idle. The first fix is a one-line change. To test it, sweep `gllm_set_attn_tile_min_span` at c1 for 6k, 31k and 88k, and check that c8 and c16 do not regress. ## Acceptance At 350 W on Nemotron, c1 decode at 6k context is at or below v0.54.1's 6.63 ms per token. The v0.54.2 gains at 31k and 88k, and under batched decode, are kept.
Sign in to join this conversation.
No labels
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
brooktrails/gllm#131
No description provided.