perf(cuda): tile long decode spans across blocks #127
Loading…
Reference in a new issue
No description provided.
Delete branch "feat/decode-span-tiling"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Speeds up decode attention at long context by splitting each KV span across many blocks when a launch would otherwise leave most of the GPU idle. Refs #124.
Problem
paged_attention_flash_kernellaunches one block per (query token, head), and that block's 8 warps walk the whole KV span. A launch therefore takes as long as its longest span, however few blocks it has. Decoding one sequence on a 32-head model launches 32 blocks, which leaves most of the GPU idle while those blocks cross the context. That is why decode time per token grew so steeply with context, and why concurrency 8 cost about the same per step as concurrency 1.Change
grid.z). Each tile writes its unnormalized online-softmax state (acc, m, l) to scratch.exp(m_t - m). Empty tiles carry m = -inf and l = 0, and contribute nothing.gllm_stage_close. It is about 1 MiB at the default target and grows only when a launch needs more. It is not taken from the bump arena, because one launch per layer would wrap the arena, and wait for the device, every step. If allocation fails, the launch runs untiled.GLLM_ATTN_TILE_BLOCKSoverrides the target for A/B runs, and a value below 2 disables tiling. The test hookssetAttnTileBlocks,setAttnTileMinSpanandattnTiledLaunchesfollow the existing dispatch hooks.Why the tiled kernel is a separate copy. Folding tiling into the existing kernel made untiled prefill about 7.5% slower, as a runtime branch and as a template instantiated both ways. That happened even though the untiled path's SASS was identical to main's. Measured on a Granite 7k prefill at 350 W with alternating builds: nsys showed the flash kernel itself at 259 vs 239 ms per launch. A separate kernel, defined before or after the original, runs at main's speed. So the original kernel is byte-for-byte unchanged, and the tiled kernel duplicates its loop, with a comment asking to keep the two in step.
Results
RTX PRO 6000 Blackwell, 350 W cap. Main and this branch were run back to back with the same prompts, 300-token replies. Values are ms per token.
Numerics
The tiles change the order in which positions are summed, so outputs differ from main in the last bits, as the warp split already does. The tile count depends on how many tokens are in the launch, so a sequence's attention can round differently depending on what it is batched with (see #98 for the existing case of this).
Tests
TestCUDAPagedAttentionTilingcompares against the CPU reference for a one-position span, fewer positions than tiles, a 5000-position span, a sliding window, and mixed spans that leave some tiles empty. Each case asserts that the launch actually tiled.go test -tags cuda ./internal/backend/cuda/ ./internal/model/...passes on an RTX 3070 (sm_86) and the RTX PRO 6000 (sm_120), andgo build,go vetandgo test ./...pass.🤖 Generated with Claude Code
A flash-attention block's warps walk the whole KV span, so a launch takes as long as its longest span however few blocks it has. Decoding one sequence launched only qHeads blocks and left most of the GPU idle. When a launch has few blocks, each (token, head) now splits its span into tiles, one block per tile, and a merge kernel combines the tiles' online-softmax states. Launches with enough blocks already, such as prefill, run the existing kernel unchanged. The tiled kernel is a separate copy of the flash kernel. Folding tiling into the existing kernel, as a runtime branch or as a template instantiated both ways, made untiled prefill about 7.5% slower even with identical SASS for the untiled path; a separate kernel did not. Measured against main at a 350 W cap, ms per token: Nemotron 3 Nano c1 6k 11.3 -> 6.6 31k 30.5 -> 8.0 88k 75.3 -> 11.9 c8 6k 22.6 -> 20.0 Granite 4.2 8b c1 7k 52.3 -> 17.4 15k 94.6 -> 22.6 31k 179.1 -> 32.0 c8 7k 56.6 -> 47.9 Prefill time is unchanged within run-to-run noise (Granite 7k, alternating builds: 10.4-10.8 s on both). GLLM_ATTN_TILE_BLOCKS overrides the block target (default 2048; below 2 disables tiling) for A/B runs. Refs #124 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>Automated review by pr-reviewer v0.54.0 | Safety Check | Nemotron 3 Nano | tracking id
r-e96446-cd2e4aThis is an AI-gene2ated review and may contain mistakes.
Status: ❌ Failed
Review failed. Tracking id
r-e96446-cd2e4a— see logs for details.Comment
@pr-reviewer-bot retryto try again.