GQA decode attention: c1 short-context decode 19% slower on Nemotron since v0.54.2 #131
Loading…
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Summary
v0.54.2 (the GQA-aware decode attention from #129) decodes Nemotron 3 Nano about 19% slower than v0.54.1 for a single request at short context: 7.86 ms per token against 6.63 at a 6,378-token prompt. Longer contexts and batched decode got faster, so the loss is confined to small launches at short span. Single-user chats at a few thousand tokens are a common case, so this is worth recovering.
Measurements
RTX PRO 6000 Blackwell (
GPU-bc3460ab-6ad9-c6ef-c955-6922515f2b7f), power capped at 350 W, no other process on the card. Both binaries were built withmake cudafrom the release tags. Model: NVIDIA-Nemotron-3-Nano-30B-A3B-NVFP4 (32 query heads, 2 KV heads, GQA group 16).Short-context repeat: one request at a time with a 6,378-token prompt, 300 completion tokens per request, and ms/token measured from the 11th token to the last. Ten requests per version, with the two servers alternated over two rounds of five:
The two rounds agree to within 0.03 ms, so this is not drift.
Same-session sweep with one run per point. Decode is measured after every request in the batch has received its first token.
Granite 4.2 8b (group 4, 8 KV heads) shows no regression at c1 7k (17.6 vs 16.9).
Likely cause (not yet profiled)
The tile count comes from
attn_tiles(numTokens, kvHeads, maxSpan), which takes the smaller of two values:ATTN_TILE_BLOCKS / (numTokens * kvHeads)blocks, andmaxSpan / ATTN_TILE_MIN_SPAN(512).For Nemotron at c1 with a 6,378-token span:
That is 24 blocks on a GPU with roughly 188 SMs. The minimum-span limit binds before the block target does. Each GQA block now does
grouptimes the work per position, yet the 512-position minimum is unchanged.At 31k the same limit gives 60 × 2 = 120 blocks, which is where the two versions break even. At 88k it gives 342 blocks, and v0.54.2 is ahead there.
Possible fixes
maxSpan * group / ATTN_TILE_MIN_SPAN, or lower the minimum span on the GQA path.numTokens * kvHeads * tileswould leave most SMs idle.The first fix is a one-line change. To test it, sweep
gllm_set_attn_tile_min_spanat c1 for 6k, 31k and 88k, and check that c8 and c16 do not regress.Acceptance
At 350 W on Nemotron, c1 decode at 6k context is at or below v0.54.1's 6.63 ms per token. The v0.54.2 gains at 31k and 88k, and under batched decode, are kept.