perf(cuda): tile long decode spans across blocks #127

Merged
rcsheets merged 1 commit from feat/decode-span-tiling into main 2026-11-03 04:37:05 +00:00
Owner

Speeds up decode attention at long context by splitting each KV span across many blocks when a launch would otherwise leave most of the GPU idle. Refs #124.

Problem

paged_attention_flash_kernel launches one block per (query token, head), and that block's 8 warps walk the whole KV span. A launch therefore takes as long as its longest span, however few blocks it has. Decoding one sequence on a 32-head model launches 32 blocks, which leaves most of the GPU idle while those blocks cross the context. That is why decode time per token grew so steeply with context, and why concurrency 8 cost about the same per step as concurrency 1.

Change

  • Tiled kernel. When a launch has few blocks, each (token, head) is split into tiles over contiguous ranges of its span, one block per tile (grid.z). Each tile writes its unnormalized online-softmax state (acc, m, l) to scratch.
  • Merge kernel. A second launch combines the tiles with the same rescale the warps already use, exp(m_t - m). Empty tiles carry m = -inf and l = 0, and contribute nothing.
  • Tile count. Enough tiles to reach a block target (default 2048), with no tile shorter than 512 positions, capped at the grid-dimension limit. Prefill launches already have enough blocks, so they stay at one tile and run the existing kernel unchanged.
  • Scratch. One persistent buffer per device, kept with the staging arena and freed in gllm_stage_close. It is about 1 MiB at the default target and grows only when a launch needs more. It is not taken from the bump arena, because one launch per layer would wrap the arena, and wait for the device, every step. If allocation fails, the launch runs untiled.
  • Knobs. GLLM_ATTN_TILE_BLOCKS overrides the target for A/B runs, and a value below 2 disables tiling. The test hooks setAttnTileBlocks, setAttnTileMinSpan and attnTiledLaunches follow the existing dispatch hooks.

Why the tiled kernel is a separate copy. Folding tiling into the existing kernel made untiled prefill about 7.5% slower, as a runtime branch and as a template instantiated both ways. That happened even though the untiled path's SASS was identical to main's. Measured on a Granite 7k prefill at 350 W with alternating builds: nsys showed the flash kernel itself at 259 vs 239 ms per launch. A separate kernel, defined before or after the original, runs at main's speed. So the original kernel is byte-for-byte unchanged, and the tiled kernel duplicates its loop, with a comment asking to keep the two in step.

Results

RTX PRO 6000 Blackwell, 350 W cap. Main and this branch were run back to back with the same prompts, 300-token replies. Values are ms per token.

Model Concurrency Context main branch
Nemotron 3 Nano NVFP4 1 6k 11.3 6.6
Nemotron 3 Nano NVFP4 1 31k 30.5 8.0
Nemotron 3 Nano NVFP4 1 88k 75.3 11.9
Nemotron 3 Nano NVFP4 8 6k 22.6 20.0
Granite 4.2 8b 1 7k 52.3 17.4
Granite 4.2 8b 1 15k 94.6 22.6
Granite 4.2 8b 1 31k 179.1 32.0
Granite 4.2 8b 8 7k 56.6 47.9
  • Prefill: time to first token is unchanged within run-to-run noise. Granite at 7k measured 10.4-10.8 s on both builds across three alternating rounds, and single runs at 15k-88k are within 0.5-3.6% of main.
  • Block target: at 512, 2048 and 8192 the results were the same within 0.3 ms per token.
  • Reference: vLLM 0.33.0 on the same card and cap measures 3.2-3.3 ms per token for Nemotron and 12.9-15.5 for Granite over the same contexts, so a gap remains at concurrency 1.

Numerics

The tiles change the order in which positions are summed, so outputs differ from main in the last bits, as the warp split already does. The tile count depends on how many tokens are in the launch, so a sequence's attention can round differently depending on what it is batched with (see #98 for the existing case of this).

Tests

  • New: TestCUDAPagedAttentionTiling compares against the CPU reference for a one-position span, fewer positions than tiles, a 5000-position span, a sliding window, and mixed spans that leave some tiles empty. Each case asserts that the launch actually tiled.
  • Runs: go test -tags cuda ./internal/backend/cuda/ ./internal/model/... passes on an RTX 3070 (sm_86) and the RTX PRO 6000 (sm_120), and go build, go vet and go test ./... pass.
  • Unrelated: one earlier run hit the intermittent leak-test failure tracked in #126. It has not recurred in seven runs since.

🤖 Generated with Claude Code

Speeds up decode attention at long context by splitting each KV span across many blocks when a launch would otherwise leave most of the GPU idle. Refs #124. ## Problem `paged_attention_flash_kernel` launches one block per (query token, head), and that block's 8 warps walk the whole KV span. A launch therefore takes as long as its longest span, however few blocks it has. Decoding one sequence on a 32-head model launches 32 blocks, which leaves most of the GPU idle while those blocks cross the context. That is why decode time per token grew so steeply with context, and why concurrency 8 cost about the same per step as concurrency 1. ## Change - **Tiled kernel.** When a launch has few blocks, each (token, head) is split into tiles over contiguous ranges of its span, one block per tile (`grid.z`). Each tile writes its unnormalized online-softmax state (acc, m, l) to scratch. - **Merge kernel.** A second launch combines the tiles with the same rescale the warps already use, `exp(m_t - m)`. Empty tiles carry m = -inf and l = 0, and contribute nothing. - **Tile count.** Enough tiles to reach a block target (default 2048), with no tile shorter than 512 positions, capped at the grid-dimension limit. Prefill launches already have enough blocks, so they stay at one tile and run the existing kernel unchanged. - **Scratch.** One persistent buffer per device, kept with the staging arena and freed in `gllm_stage_close`. It is about 1 MiB at the default target and grows only when a launch needs more. It is not taken from the bump arena, because one launch per layer would wrap the arena, and wait for the device, every step. If allocation fails, the launch runs untiled. - **Knobs.** `GLLM_ATTN_TILE_BLOCKS` overrides the target for A/B runs, and a value below 2 disables tiling. The test hooks `setAttnTileBlocks`, `setAttnTileMinSpan` and `attnTiledLaunches` follow the existing dispatch hooks. **Why the tiled kernel is a separate copy.** Folding tiling into the existing kernel made untiled prefill about 7.5% slower, as a runtime branch and as a template instantiated both ways. That happened even though the untiled path's SASS was identical to main's. Measured on a Granite 7k prefill at 350 W with alternating builds: nsys showed the flash kernel itself at 259 vs 239 ms per launch. A separate kernel, defined before or after the original, runs at main's speed. So the original kernel is byte-for-byte unchanged, and the tiled kernel duplicates its loop, with a comment asking to keep the two in step. ## Results RTX PRO 6000 Blackwell, 350 W cap. Main and this branch were run back to back with the same prompts, 300-token replies. Values are ms per token. | Model | Concurrency | Context | main | branch | |---|---|---|---|---| | Nemotron 3 Nano NVFP4 | 1 | 6k | 11.3 | 6.6 | | Nemotron 3 Nano NVFP4 | 1 | 31k | 30.5 | 8.0 | | Nemotron 3 Nano NVFP4 | 1 | 88k | 75.3 | 11.9 | | Nemotron 3 Nano NVFP4 | 8 | 6k | 22.6 | 20.0 | | Granite 4.2 8b | 1 | 7k | 52.3 | 17.4 | | Granite 4.2 8b | 1 | 15k | 94.6 | 22.6 | | Granite 4.2 8b | 1 | 31k | 179.1 | 32.0 | | Granite 4.2 8b | 8 | 7k | 56.6 | 47.9 | - **Prefill:** time to first token is unchanged within run-to-run noise. Granite at 7k measured 10.4-10.8 s on both builds across three alternating rounds, and single runs at 15k-88k are within 0.5-3.6% of main. - **Block target:** at 512, 2048 and 8192 the results were the same within 0.3 ms per token. - **Reference:** vLLM 0.33.0 on the same card and cap measures 3.2-3.3 ms per token for Nemotron and 12.9-15.5 for Granite over the same contexts, so a gap remains at concurrency 1. ## Numerics The tiles change the order in which positions are summed, so outputs differ from main in the last bits, as the warp split already does. The tile count depends on how many tokens are in the launch, so a sequence's attention can round differently depending on what it is batched with (see #98 for the existing case of this). ## Tests - **New:** `TestCUDAPagedAttentionTiling` compares against the CPU reference for a one-position span, fewer positions than tiles, a 5000-position span, a sliding window, and mixed spans that leave some tiles empty. Each case asserts that the launch actually tiled. - **Runs:** `go test -tags cuda ./internal/backend/cuda/ ./internal/model/...` passes on an RTX 3070 (sm_86) and the RTX PRO 6000 (sm_120), and `go build`, `go vet` and `go test ./...` pass. - **Unrelated:** one earlier run hit the intermittent leak-test failure tracked in #126. It has not recurred in seven runs since. 🤖 Generated with [Claude Code](https://claude.com/claude-code)
perf(cuda): tile long decode spans across blocks
All checks were successful
ci / test_and_build (pull_request) Successful in 1m57s
0a2943a76b
A flash-attention block's warps walk the whole KV span, so a launch takes as
long as its longest span however few blocks it has. Decoding one sequence
launched only qHeads blocks and left most of the GPU idle. When a launch has
few blocks, each (token, head) now splits its span into tiles, one block per
tile, and a merge kernel combines the tiles' online-softmax states. Launches
with enough blocks already, such as prefill, run the existing kernel unchanged.

The tiled kernel is a separate copy of the flash kernel. Folding tiling into
the existing kernel, as a runtime branch or as a template instantiated both
ways, made untiled prefill about 7.5% slower even with identical SASS for the
untiled path; a separate kernel did not.

Measured against main at a 350 W cap, ms per token:

  Nemotron 3 Nano  c1  6k 11.3 -> 6.6   31k 30.5 -> 8.0   88k 75.3 -> 11.9
                   c8  6k 22.6 -> 20.0
  Granite 4.2 8b   c1  7k 52.3 -> 17.4  15k 94.6 -> 22.6  31k 179.1 -> 32.0
                   c8  7k 56.6 -> 47.9

Prefill time is unchanged within run-to-run noise (Granite 7k, alternating
builds: 10.4-10.8 s on both).

GLLM_ATTN_TILE_BLOCKS overrides the block target (default 2048; below 2
disables tiling) for A/B runs.

Refs #124

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Collaborator

Automated review by pr-reviewer v0.54.0 | Safety Check | Nemotron 3 Nano | tracking id r-e96446-cd2e4a
This is an AI-gene2ated review and may contain mistakes.

Status: ❌ Failed


Review failed. Tracking id r-e96446-cd2e4a — see logs for details.

Comment @pr-reviewer-bot retry to try again.

<!-- pr-reviewer:review --> *Automated review by [pr-reviewer](https://git.brooktrails.org/brooktrails/pr-reviewer) v0.54.0 | Safety Check | Nemotron 3 Nano | tracking id `r-e96446-cd2e4a`* *This is an AI-gene2ated review and may contain mistakes.* **Status:** ❌ Failed --- Review failed. Tracking id `r-e96446-cd2e4a` — see logs for details. Comment `@pr-reviewer-bot retry` to try again.
rcsheets deleted branch feat/decode-span-tiling 2026-11-03 04:37:05 +00:00
Sign in to join this conversation.
No reviewers
No labels
No milestone
No project
No assignees
2 participants
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
brooktrails/gllm!127
No description provided.