perf(cuda): make decode attention GQA-aware #129
Loading…
Reference in a new issue
No description provided.
Delete branch "feat/attn-gqa"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
What
The CUDA decode attention kernel launched one block per (token, query head), so every query head in a GQA group read the same K/V rows. On Nemotron 3 (32 query heads over 2 KV heads) that was 16x the KV bytes the model needs; on Granite 4.2 8b it was 4x.
Both flash bodies now own one (token, KV head) per block and compute its whole group:
headWayshead slices xposWaysposition slices;headWaysis the smallest power of two keeping a warp withinATTN_MAX_HEADS_PER_WARP(default 4) heads, so MHA keeps the old shape exactlyATTN_WARPS * ATTN_MAX_HEADS_PER_WARP(32 at the defaults) falls back to the scalar kernelNV, the float4 depth of a head, so the per-head register arrays have compile-time bounds and the predecessor's local-memory spill ofqv[]is goneReview pass
A follow-up commit tightens three edges found on review:
hpwinstead of running allATTN_MAX_HEADS_PER_WARPpasses, so an ungrouped model pays two barriers per head instead of eightstatic_assert(ATTN_MAX_VEC == 4)at the NV dispatch, so a future wider register array cannot silently run heads with their upper lanes truncatedThe tests now exercise every NV the launcher instantiates: 256 (NV=2) and 384 (NV=3) join the existing 64/128 and 512.
Validation
TestCUDAPagedAttentionGroupscovers group sizes 1, 3, 5, 6 and 16 in both the untiled and tiled bodies, plus a group-40 shape that must fall back, comparing every result to the CPU reference and asserting the launch counters.attn_bench_cuda_test.goadds the issue's two workloads as a benchmark.go build ./...,go vet ./...,go test ./..., andgo test -tags cuda ./internal/backend/cuda/pass;gofmtis clean.ncuon the issue's workload (16 sequences x 30k context, Nemotron 3, one launch), idle card:That is 3.4x on the kernel, with K/V traffic now the ideal 983 MB for the launch. End to end, host staging included, the benchmark goes from 4.72 to 1.41 ms per launch; Granite 4.2 8b's 4-way group goes from 0.62 to 0.35 ms (1.8x), because L2 already served part of the duplicate traffic at group 4. The kernel is now latency-bound (58% DRAM and SM, 1.2 eligible warps per scheduler); per-warp instruction-level parallelism is the next lever.
Closes #128.
Automated review by pr-reviewer v0.54.0 | Safety Check | Nemotron 3 Nano | tracking id
r-e99a37-1d2732This is an AI-generated review and may contain mistakes.
Status: ❌ Failed
Review failed. Tracking id
r-e99a37-1d2732— see logs for details.Comment
@pr-reviewer-bot retryto try again.