Decode attention re-reads K/V once per query head: make it GQA-aware #128

Closed
opened 2026-11-03 06:14:31 +00:00 by rcsheets · 0 comments
Owner

Batched decode at long context is now limited by the decode attention kernel reading each KV head once per query head. On grouped-query models that repeats the same K/V reads 4x (Granite 4.2 8b) to 16x (Nemotron 3 Nano).

Evidence

Measured on 0.54.1 (with span tiling from #127), RTX PRO 6000 Blackwell, 350 W cap.

  • Current workload. A reasoning experiment running Nemotron 3 Nano NVFP4 at 16 concurrent sequences, with 4.4k-token prompts and replies up to 65k tokens, decodes at about 68 ms per step, roughly 235 tokens/s in total. A 65k-token reply takes about 75 minutes.
  • Attention's share, estimated. At concurrency 1, step time rises from 6.6 ms at 6k context to 11.9 ms at 88k, about 0.065 ms per 1k context tokens. Scaled to 16 sequences averaging about 35k context, that is about 36 ms of a 68 ms step. This is an estimate from the slope, not a profile.
  • Distance from bandwidth. Nemotron stores 12 KiB of KV per token in float32 (6 attention layers x 2 KV heads x 128 dims x K and V x 4 bytes). Reading it once for 16 x 35k tokens is 6.9 GB, about 3.8 ms at the card's 1.8 TB/s, so the estimated 36 ms is roughly 9x off that bound.
  • Reference. vLLM 0.33.0 on the same card decodes Nemotron at 3.2-3.3 ms per token at concurrency 1 from 6k to 88k context, flat with context (#124). Its log shows the FlashInfer XQA decode backend built with head_group_ratio_16, which processes a KV head's whole query group together.

Cause

paged_attention_flash_kernel and its tiled variant launch one block per (token, query head) and map query head h to KV head h / group. Each of a group's query-head blocks loads the same K and V rows independently. With Nemotron's 32 query heads over 2 KV heads, every K/V row is fetched 16 times per step, from L2 at best and DRAM at worst.

Proposed change

Make decode attention GQA-aware: one block, or one tile, per (token, KV head), which loads each K/V row once and computes the dot products and running softmax state for all group query heads that share it. Per-lane registers hold group partial accumulators. Where that is too many for 16-way groups, split the group across two to four warps' worth of state. The span tiling and merge from #127 carry over unchanged, with tiles per (token, KV head).

Expected effect: K/V traffic per step drops by the group factor. For Nemotron that should bring batched long-context attention near the bandwidth figure above. The size of the gain needs measuring, since part of the current cost may be L2-served.

  • An 8-bit KV cache would cut the remaining bytes 4x. vLLM runs E4M3 KV here. It changes numerics, so it belongs in its own issue, after this one.
  • #123 (prefill) does not affect this workload: its prompts are 4.4k tokens and prefill takes about 2 s.

How to verify

Profile a 16-way decode at about 30k context with nsys, before and after. Check that the flash kernel's share drops and step time falls, and that greedy outputs still match the CPU reference within the existing attention tests.

Batched decode at long context is now limited by the decode attention kernel reading each KV head once per query head. On grouped-query models that repeats the same K/V reads 4x (Granite 4.2 8b) to 16x (Nemotron 3 Nano). ## Evidence Measured on 0.54.1 (with span tiling from #127), RTX PRO 6000 Blackwell, 350 W cap. - **Current workload.** A reasoning experiment running Nemotron 3 Nano NVFP4 at 16 concurrent sequences, with 4.4k-token prompts and replies up to 65k tokens, decodes at about 68 ms per step, roughly 235 tokens/s in total. A 65k-token reply takes about 75 minutes. - **Attention's share, estimated.** At concurrency 1, step time rises from 6.6 ms at 6k context to 11.9 ms at 88k, about 0.065 ms per 1k context tokens. Scaled to 16 sequences averaging about 35k context, that is about 36 ms of a 68 ms step. This is an estimate from the slope, not a profile. - **Distance from bandwidth.** Nemotron stores 12 KiB of KV per token in float32 (6 attention layers x 2 KV heads x 128 dims x K and V x 4 bytes). Reading it once for 16 x 35k tokens is 6.9 GB, about 3.8 ms at the card's 1.8 TB/s, so the estimated 36 ms is roughly 9x off that bound. - **Reference.** vLLM 0.33.0 on the same card decodes Nemotron at 3.2-3.3 ms per token at concurrency 1 from 6k to 88k context, flat with context (#124). Its log shows the FlashInfer XQA decode backend built with `head_group_ratio_16`, which processes a KV head's whole query group together. ## Cause `paged_attention_flash_kernel` and its tiled variant launch one block per (token, query head) and map query head `h` to KV head `h / group`. Each of a group's query-head blocks loads the same K and V rows independently. With Nemotron's 32 query heads over 2 KV heads, every K/V row is fetched 16 times per step, from L2 at best and DRAM at worst. ## Proposed change Make decode attention GQA-aware: one block, or one tile, per (token, KV head), which loads each K/V row once and computes the dot products and running softmax state for all `group` query heads that share it. Per-lane registers hold `group` partial accumulators. Where that is too many for 16-way groups, split the group across two to four warps' worth of state. The span tiling and merge from #127 carry over unchanged, with tiles per (token, KV head). Expected effect: K/V traffic per step drops by the group factor. For Nemotron that should bring batched long-context attention near the bandwidth figure above. The size of the gain needs measuring, since part of the current cost may be L2-served. ## Related, separate - An 8-bit KV cache would cut the remaining bytes 4x. vLLM runs E4M3 KV here. It changes numerics, so it belongs in its own issue, after this one. - #123 (prefill) does not affect this workload: its prompts are 4.4k tokens and prefill takes about 2 s. ## How to verify Profile a 16-way decode at about 30k context with nsys, before and after. Check that the flash kernel's share drops and step time falls, and that greedy outputs still match the CPU reference within the existing attention tests.
Sign in to join this conversation.
No labels
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
brooktrails/gllm#128
No description provided.