Decode attention re-reads K/V once per query head: make it GQA-aware #128
Loading…
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Batched decode at long context is now limited by the decode attention kernel reading each KV head once per query head. On grouped-query models that repeats the same K/V reads 4x (Granite 4.2 8b) to 16x (Nemotron 3 Nano).
Evidence
Measured on 0.54.1 (with span tiling from #127), RTX PRO 6000 Blackwell, 350 W cap.
head_group_ratio_16, which processes a KV head's whole query group together.Cause
paged_attention_flash_kerneland its tiled variant launch one block per (token, query head) and map query headhto KV headh / group. Each of a group's query-head blocks loads the same K and V rows independently. With Nemotron's 32 query heads over 2 KV heads, every K/V row is fetched 16 times per step, from L2 at best and DRAM at worst.Proposed change
Make decode attention GQA-aware: one block, or one tile, per (token, KV head), which loads each K/V row once and computes the dot products and running softmax state for all
groupquery heads that share it. Per-lane registers holdgrouppartial accumulators. Where that is too many for 16-way groups, split the group across two to four warps' worth of state. The span tiling and merge from #127 carry over unchanged, with tiles per (token, KV head).Expected effect: K/V traffic per step drops by the group factor. For Nemotron that should bring batched long-context attention near the bandwidth figure above. The size of the gain needs measuring, since part of the current cost may be L2-served.
Related, separate
How to verify
Profile a 16-way decode at about 30k context with nsys, before and after. Check that the flash kernel's share drops and step time falls, and that greedy outputs still match the CPU reference within the existing attention tests.