perf(cuda): make decode attention GQA-aware #129

Merged
rcsheets merged 4 commits from feat/attn-gqa into main 2026-11-03 08:27:01 +00:00
Owner

What

The CUDA decode attention kernel launched one block per (token, query head), so every query head in a GQA group read the same K/V rows. On Nemotron 3 (32 query heads over 2 KV heads) that was 16x the KV bytes the model needs; on Granite 4.2 8b it was 4x.

Both flash bodies now own one (token, KV head) per block and compute its whole group:

  • the block's 8 warps split into headWays head slices x posWays position slices; headWays is the smallest power of two keeping a warp within ATTN_MAX_HEADS_PER_WARP (default 4) heads, so MHA keeps the old shape exactly
  • warps sharing a head slice stride the span together; warps sharing a position read the same row inside one block, turning the duplicate fetch into an L1 hit instead of DRAM/L2 traffic
  • a group wider than ATTN_WARPS * ATTN_MAX_HEADS_PER_WARP (32 at the defaults) falls back to the scalar kernel
  • span tiling (#127) carries over, now with tiles per (token, KV head); the merge kernel stays per query head because its scratch is indexed that way
  • both kernels are templated on NV, the float4 depth of a head, so the per-head register arrays have compile-time bounds and the predecessor's local-memory spill of qv[] is gone

Review pass

A follow-up commit tightens three edges found on review:

  • the combine loop stops at hpw instead of running all ATTN_MAX_HEADS_PER_WARP passes, so an ungrouped model pays two barriers per head instead of eight
  • static_assert(ATTN_MAX_VEC == 4) at the NV dispatch, so a future wider register array cannot silently run heads with their upper lanes truncated
  • tile-scratch allocation failures are counted and warned on the first one; the scratch is allocated outside the memory plan and now scales with the group, and a failure quietly runs untiled

The tests now exercise every NV the launcher instantiates: 256 (NV=2) and 384 (NV=3) join the existing 64/128 and 512.

Validation

TestCUDAPagedAttentionGroups covers group sizes 1, 3, 5, 6 and 16 in both the untiled and tiled bodies, plus a group-40 shape that must fall back, comparing every result to the CPU reference and asserting the launch counters. attn_bench_cuda_test.go adds the issue's two workloads as a benchmark.

go build ./..., go vet ./..., go test ./..., and go test -tags cuda ./internal/backend/cuda/ pass; gofmt is clean.

ncu on the issue's workload (16 sequences x 30k context, Nemotron 3, one launch), idle card:

main this
tiled flash kernel 3.39 ms 1.00 ms
DRAM throughput 26.6% 58.3%
achieved occupancy 75.8% 56.9%

That is 3.4x on the kernel, with K/V traffic now the ideal 983 MB for the launch. End to end, host staging included, the benchmark goes from 4.72 to 1.41 ms per launch; Granite 4.2 8b's 4-way group goes from 0.62 to 0.35 ms (1.8x), because L2 already served part of the duplicate traffic at group 4. The kernel is now latency-bound (58% DRAM and SM, 1.2 eligible warps per scheduler); per-warp instruction-level parallelism is the next lever.

Closes #128.

## What The CUDA decode attention kernel launched one block per (token, query head), so every query head in a GQA group read the same K/V rows. On Nemotron 3 (32 query heads over 2 KV heads) that was 16x the KV bytes the model needs; on Granite 4.2 8b it was 4x. Both flash bodies now own one (token, **KV head**) per block and compute its whole group: - the block's 8 warps split into `headWays` head slices x `posWays` position slices; `headWays` is the smallest power of two keeping a warp within `ATTN_MAX_HEADS_PER_WARP` (default 4) heads, so MHA keeps the old shape exactly - warps sharing a head slice stride the span together; warps sharing a position read the same row inside one block, turning the duplicate fetch into an L1 hit instead of DRAM/L2 traffic - a group wider than `ATTN_WARPS * ATTN_MAX_HEADS_PER_WARP` (32 at the defaults) falls back to the scalar kernel - span tiling (#127) carries over, now with tiles per (token, KV head); the merge kernel stays per query head because its scratch is indexed that way - both kernels are templated on `NV`, the float4 depth of a head, so the per-head register arrays have compile-time bounds and the predecessor's local-memory spill of `qv[]` is gone ## Review pass A follow-up commit tightens three edges found on review: - the combine loop stops at `hpw` instead of running all `ATTN_MAX_HEADS_PER_WARP` passes, so an ungrouped model pays two barriers per head instead of eight - `static_assert(ATTN_MAX_VEC == 4)` at the NV dispatch, so a future wider register array cannot silently run heads with their upper lanes truncated - tile-scratch allocation failures are counted and warned on the first one; the scratch is allocated outside the memory plan and now scales with the group, and a failure quietly runs untiled The tests now exercise every NV the launcher instantiates: 256 (NV=2) and 384 (NV=3) join the existing 64/128 and 512. ## Validation `TestCUDAPagedAttentionGroups` covers group sizes 1, 3, 5, 6 and 16 in both the untiled and tiled bodies, plus a group-40 shape that must fall back, comparing every result to the CPU reference and asserting the launch counters. `attn_bench_cuda_test.go` adds the issue's two workloads as a benchmark. `go build ./...`, `go vet ./...`, `go test ./...`, and `go test -tags cuda ./internal/backend/cuda/` pass; `gofmt` is clean. `ncu` on the issue's workload (16 sequences x 30k context, Nemotron 3, one launch), idle card: | | main | this | | --- | --- | --- | | tiled flash kernel | 3.39 ms | 1.00 ms | | DRAM throughput | 26.6% | 58.3% | | achieved occupancy | 75.8% | 56.9% | That is 3.4x on the kernel, with K/V traffic now the ideal 983 MB for the launch. End to end, host staging included, the benchmark goes from 4.72 to 1.41 ms per launch; Granite 4.2 8b's 4-way group goes from 0.62 to 0.35 ms (1.8x), because L2 already served part of the duplicate traffic at group 4. The kernel is now latency-bound (58% DRAM and SM, 1.2 eligible warps per scheduler); per-warp instruction-level parallelism is the next lever. Closes #128.
The flash decode kernel owned one (token, query head) per block, so on a
grouped-query model every head in a group launched its own block and read
the same K/V rows. The cached traffic the kernel is bound on was therefore
the model's KV bytes times the group factor -- 16x on Nemotron 3 (32 query
heads over 2 KV heads), 4x on Granite 4.2 8b.

Both flash bodies now own one (token, KV head) and compute its whole group.
A warp cannot hold a 16-way group in registers without spilling, so the
block's 8 warps split into headWays head slices by posWays position slices:
headWays is the smallest power of two keeping a warp at or below
ATTN_MAX_HEADS_PER_WARP (default 4) heads, so group 1 keeps the old shape.
Warps sharing a head slice stride the span together; warps sharing a
position read the same row within one block, turning the duplicate fetch
into an L1 hit. A group wider than ATTN_WARPS*ATTN_MAX_HEADS_PER_WARP (32)
falls back to the scalar kernel.

The tiled body follows: one tile set per (token, KV head), still merged by
the per-query-head merge kernel. Both kernels are templated on NV, the
float4 depth of a head, so the per-head register arrays have compile-time
bounds and the predecessor's local-memory spill of qv[] goes away.

ncu on 16 sequences x 30k context, one launch (3.72 ms -> 1.03 ms, 3.6x;
DRAM throughput 22.9% -> 59.5%) -- a ratio only, since a separate gllm
server shared the card during the measurement. TestCUDAPagedAttentionGroups
pins groups 1, 3, 5, 6 and 16 in both bodies plus the group-40 fallback
against the CPU reference.

Refs #128.
test(cuda): benchmark long-context decode attention
All checks were successful
ci / test_and_build (pull_request) Successful in 42s
0b0275fb0a
Adds the issue's two workloads as a benchmark: Nemotron 3 Nano's 16-way
group (16 sequences x 30k context) and Granite 4.2 8b's 4-way group (8 x
7k), one query token per sequence. Each sequence owns its own blocks so L2
cannot serve one sequence's traffic to another. GB/s is reported against
the bytes one pass over the visible rows must read, the floor a
grouped-query-aware kernel can reach; ms/launch tracks the issue's
per-step evidence.

Refs #128.
Collaborator

Automated review by pr-reviewer v0.54.0 | Safety Check | Nemotron 3 Nano | tracking id r-e99a37-1d2732
This is an AI-generated review and may contain mistakes.

Status: ❌ Failed


Review failed. Tracking id r-e99a37-1d2732 — see logs for details.

Comment @pr-reviewer-bot retry to try again.

<!-- pr-reviewer:review --> *Automated review by [pr-reviewer](https://git.brooktrails.org/brooktrails/pr-reviewer) v0.54.0 | Safety Check | Nemotron 3 Nano | tracking id `r-e99a37-1d2732`* *This is an AI-generated review and may contain mistakes.* **Status:** ❌ Failed --- Review failed. Tracking id `r-e99a37-1d2732` — see logs for details. Comment `@pr-reviewer-bot retry` to try again.
perf(cuda): tighten GQA decode attention's dispatch, combine and fallback
All checks were successful
ci / test_and_build (pull_request) Successful in 38s
88c855451c
- Stop the flash kernels' combine loop at hpw. It ran
  ATTN_MAX_HEADS_PER_WARP passes of two barriers each, so an ungrouped
  model paid eight barriers for its one head.
- Static-assert ATTN_MAX_VEC == 4 at the NV dispatch. The switches
  instantiate NV 1..4 and send anything else to 4, so a deeper
  ATTN_MAX_VEC without new cases would drop a wide head's upper lanes
  and return wrong output with no error.
- Count tile-scratch allocation failures and warn on the first one. The
  scratch is allocated lazily outside the memory plan and now scales
  with the GQA group (about 17 MB at Nemotron 3's shape, up from about
  1 MB); a failed allocation runs untiled, which on a full card could
  silently drop tiling for the life of the process.
- Cover the NV=2 instantiation: TestCUDAPagedAttentionGroups repeats
  groups 1 and 16 at headDim 256.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
test(cuda): cover the NV=3 attention instantiation
All checks were successful
ci / test_and_build (pull_request) Successful in 41s
516b7661ed
The launcher instantiates the flash kernels at NV 1..4, but the tests only
reached 1, 2 and 4: 64/128/512 run elsewhere and the review commit added
256. Add headDim 384 (96 float4 per head) to the head-dim boundary walk and
a group-16 case at the same width, which exercises the tiled body too. Every
dispatch arm is now run at least once.

Refs #128.
rcsheets deleted branch feat/attn-gqa 2026-11-03 08:27:01 +00:00
Sign in to join this conversation.
No reviewers
No labels
No milestone
No project
No assignees
2 participants
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
brooktrails/gllm!129
No description provided.