perf(cuda): compute several output rows per warp in the NVFP4 GEMV #102

Merged
rcsheets merged 1 commit from perf/nvfp4-gemv-rows into main 2026-10-01 08:59:10 +00:00
Owner

Problem

A profiled Nemotron 3 Nano decode step (8 concurrent requests, main at bf7b812) takes 23 ms, and the GPU is busy for 86% of it. The two wide-load NVFP4 GEMV kernels account for 70% of the step: the grouped MoE launch 8.5 ms and the single-matrix launch 7.6 ms. The per-layer MoE gate readbacks, which looked like the next target, are 0.5 ms.

A new microbenchmark shows what bounds those kernels. Launch time is linear in the number of products (activation rows × out × in), at about 1.5 × 10¹² products per second for every shape and row count. It does not follow the weight bytes: a kernel that only reads the same weights takes 2.5 µs of a 49 µs 8-row launch. Each warp computed one output row, so every product paid for a quarter of an activation load, and every warp fetched each activation row for itself. Pointing those loads at a small window, with nothing else changed, cut the 8-row launch to 27 µs.

Change

gemv_vec_rows<ROWS, OUTS> replaces gemv_vec_row<ROWS> in both the plain and the grouped kernel.

  • OUTS output rows per warp. Each lane carries OUTS weight rows through a chunk, so one activation load feeds OUTS products per column.
  • Row tiles. Row counts above 8 run as tiles of 8, each a full pass over the weights, so the accumulators keep a fixed size for any row count up to the existing 32-row limit.
  • Same arithmetic. Every (row, output) value is computed by the same operations in the same order as before, so results are bit-identical for any OUTS.
  • Choosing OUTS. More rows per warp means proportionally fewer warps, and a launch that runs short of warps slows down again. gemv_rows_per_warp picks the largest count that leaves every SM four warps. A single matrix is capped at 2 for one or two rows and at 4 otherwise; a grouped launch has a set of warps per expert and usually takes 8.
3712×2688, 8 rows, output rows per warp 1 2 4 8
launch time 49 µs 35 µs 29 µs 44 µs

Measurements

RTX PRO 6000 Blackwell, 300 W cap. Kernel times are from BenchmarkCUDANVFP4GEMV and BenchmarkCUDANVFP4GEMVGrouped, which re-read the same weights each launch and so serve them from the L2 cache; a served step reads its weights from DRAM.

launch before after
3712×2688, 1 row 8.5 µs 8.1 µs
3712×2688, 8 rows 49 µs 30 µs
3712×2688, 32 rows 254 µs 116 µs
2688×4096, 8 rows 51 µs 34 µs
grouped 1856×2688, 8 tokens over 37 experts 152 µs 112 µs
grouped 1856×2688, 32 tokens over 48 experts 554 µs 263 µs
RTX 3070: 3712×2688, 8 rows 247 µs 108 µs

Serving NVIDIA-Nemotron-3-Nano-30B-A3B-NVFP4 with --kv-cache 8GiB --max-model-len 8192, 8 concurrent chat requests × 256 tokens, greedy, measured after one warmup pass:

main this branch
output tokens per second 382.6 437.4
GPU energy per token 0.52 J 0.44 J

Verification

  • GEMV outputs over five shapes, ten row counts (1 to 32) and two grouped layouts are bit-identical between the old and new kernels, on the Blackwell and on the RTX 3070, and identical between the two GPUs. This was checked with a temporary dump test that is not part of the change.
  • Eight sequential greedy requests (four ~5k-token prefills at 64 tokens, four short chats at 128) return byte-identical replies on main and on this branch.
  • The CUDA backend test package (43 tests) and the nemotronh CUDA tests pass on the Blackwell. The NVFP4 GEMV and grouped tests pass on the RTX 3070.
  • "What is 17*23? Answer briefly." returns 391.

Not run: the GLLM_CUDA_SYNC=1 comparison (execution order is unchanged) and the full glbench matrix against vLLM.

Remaining headroom

The single-matrix launch is still short of warps: a lane walks three or four chunks serially. Giving each chunk pass its own warp and combining the partial sums in the same order would triple the warps while keeping results bit-identical. Whether the grouped launch is near the DRAM bandwidth limit in a served step is not yet measured.

🤖 Generated with Claude Code

## Problem A profiled Nemotron 3 Nano decode step (8 concurrent requests, main at bf7b812) takes 23 ms, and the GPU is busy for 86% of it. The two wide-load NVFP4 GEMV kernels account for 70% of the step: the grouped MoE launch 8.5 ms and the single-matrix launch 7.6 ms. The per-layer MoE gate readbacks, which looked like the next target, are 0.5 ms. A new microbenchmark shows what bounds those kernels. Launch time is linear in the number of products (activation rows × out × in), at about 1.5 × 10¹² products per second for every shape and row count. It does not follow the weight bytes: a kernel that only reads the same weights takes 2.5 µs of a 49 µs 8-row launch. Each warp computed one output row, so every product paid for a quarter of an activation load, and every warp fetched each activation row for itself. Pointing those loads at a small window, with nothing else changed, cut the 8-row launch to 27 µs. ## Change `gemv_vec_rows<ROWS, OUTS>` replaces `gemv_vec_row<ROWS>` in both the plain and the grouped kernel. - **OUTS output rows per warp.** Each lane carries OUTS weight rows through a chunk, so one activation load feeds OUTS products per column. - **Row tiles.** Row counts above 8 run as tiles of 8, each a full pass over the weights, so the accumulators keep a fixed size for any row count up to the existing 32-row limit. - **Same arithmetic.** Every (row, output) value is computed by the same operations in the same order as before, so results are bit-identical for any OUTS. - **Choosing OUTS.** More rows per warp means proportionally fewer warps, and a launch that runs short of warps slows down again. `gemv_rows_per_warp` picks the largest count that leaves every SM four warps. A single matrix is capped at 2 for one or two rows and at 4 otherwise; a grouped launch has a set of warps per expert and usually takes 8. | 3712×2688, 8 rows, output rows per warp | 1 | 2 | 4 | 8 | | --- | ---: | ---: | ---: | ---: | | launch time | 49 µs | 35 µs | 29 µs | 44 µs | ## Measurements RTX PRO 6000 Blackwell, 300 W cap. Kernel times are from `BenchmarkCUDANVFP4GEMV` and `BenchmarkCUDANVFP4GEMVGrouped`, which re-read the same weights each launch and so serve them from the L2 cache; a served step reads its weights from DRAM. | launch | before | after | | --- | ---: | ---: | | 3712×2688, 1 row | 8.5 µs | 8.1 µs | | 3712×2688, 8 rows | 49 µs | 30 µs | | 3712×2688, 32 rows | 254 µs | 116 µs | | 2688×4096, 8 rows | 51 µs | 34 µs | | grouped 1856×2688, 8 tokens over 37 experts | 152 µs | 112 µs | | grouped 1856×2688, 32 tokens over 48 experts | 554 µs | 263 µs | | RTX 3070: 3712×2688, 8 rows | 247 µs | 108 µs | Serving NVIDIA-Nemotron-3-Nano-30B-A3B-NVFP4 with `--kv-cache 8GiB --max-model-len 8192`, 8 concurrent chat requests × 256 tokens, greedy, measured after one warmup pass: | | main | this branch | | --- | ---: | ---: | | output tokens per second | 382.6 | 437.4 | | GPU energy per token | 0.52 J | 0.44 J | ## Verification - GEMV outputs over five shapes, ten row counts (1 to 32) and two grouped layouts are bit-identical between the old and new kernels, on the Blackwell and on the RTX 3070, and identical between the two GPUs. This was checked with a temporary dump test that is not part of the change. - Eight sequential greedy requests (four ~5k-token prefills at 64 tokens, four short chats at 128) return byte-identical replies on main and on this branch. - The CUDA backend test package (43 tests) and the `nemotronh` CUDA tests pass on the Blackwell. The NVFP4 GEMV and grouped tests pass on the RTX 3070. - "What is 17*23? Answer briefly." returns 391. Not run: the `GLLM_CUDA_SYNC=1` comparison (execution order is unchanged) and the full glbench matrix against vLLM. ## Remaining headroom The single-matrix launch is still short of warps: a lane walks three or four chunks serially. Giving each chunk pass its own warp and combining the partial sums in the same order would triple the warps while keeping results bit-identical. Whether the grouped launch is near the DRAM bandwidth limit in a served step is not yet measured. 🤖 Generated with [Claude Code](https://claude.com/claude-code)
perf(cuda): compute several output rows per warp in the NVFP4 GEMV
All checks were successful
ci / test_and_build (pull_request) Successful in 27s
c89e874fa0
A profiled Nemotron 3 Nano decode step (8 requests) spent 70% of its
23 ms in the two wide-load NVFP4 GEMV kernels, with the GPU busy 86% of
the step. A new microbenchmark (BenchmarkCUDANVFP4GEMV and
BenchmarkCUDANVFP4GEMVGrouped) showed why: launch time was linear in
the products, about 1.5e12 a second at every shape and row count, and
not in the weight bytes. A kernel that only read the same weights took
2.5 us of a 49 us 8-row launch. Each warp computed one output row, so
every product paid for a quarter of an activation load and every warp
fetched each activation row for itself; pointing those loads at a small
window alone cut the launch to 27 us.

gemv_vec_rows replaces gemv_vec_row: a warp computes K consecutive
output rows, each lane carrying K weight rows through a chunk, so one
activation load feeds K products per column. Row counts above 8 run as
tiles of 8, each a full pass over the weights, so the accumulators keep
a fixed size. Every (row, output) value is computed by the same
operations in the same order as before, so the results are
bit-identical for any K.

K rows per warp is 1/K the warps, and a launch short of warps slows
down again: 8 rows through 3712x2688 took 49 us at K = 1, 35 at 2, 29
at 4 and 44 at 8. gemv_rows_per_warp picks the largest K that leaves
every SM four warps, capped at 2 for a single matrix with one or two
rows (which gains nothing from more) and at 4 otherwise; a grouped
launch has a set of warps per expert and usually takes 8.

Measured on the RTX PRO 6000 Blackwell (300 W cap), launch times from
the benchmark, which re-reads the same weights and so serves them from
L2:

  3712x2688, 8 rows                      49 -> 30 us
  3712x2688, 32 rows                    254 -> 116 us
  2688x4096, 8 rows                      51 -> 34 us
  grouped 1856x2688, 8 tokens/37 experts 152 -> 112 us
  grouped 1856x2688, 32 tokens/48 experts 554 -> 263 us
  one row, any shape                     unchanged to 15% faster

On an RTX 3070, 3712x2688 at 8 rows went 247 -> 108 us.

Nemotron 3 Nano NVFP4, 8 concurrent requests x 256 tokens: 383 -> 437
tok/s, 0.52 -> 0.44 J/token of GPU energy. Eight sequential greedy
requests (four ~5k-token prefills, four short chats) produced
byte-identical replies before and after. GEMV outputs over five shapes,
ten row counts and two grouped layouts were bit-identical between the
old and new kernels on both GPUs, and identical between the GPUs.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Collaborator

Automated review by pr-reviewer v0.52.3 | Safety Check | Ministral 3 Instruct | tracking id r-be1e81-81ee69
This is an AI-generated review and may contain mistakes.

Status: ❌ Failed


This review couldn't be completed: that model isn't loaded on the inference service right now, and no alternate model was able to review it either. Consider splitting this PR into smaller changes. Tracking id r-be1e81-81ee69.

Comment @pr-reviewer-bot retry to try again.

<!-- pr-reviewer:review --> *Automated review by [pr-reviewer](https://git.brooktrails.org/brooktrails/pr-reviewer) v0.52.3 | Safety Check | Ministral 3 Instruct | tracking id `r-be1e81-81ee69`* *This is an AI-generated review and may contain mistakes.* **Status:** ❌ Failed --- This review couldn't be completed: that model isn't loaded on the inference service right now, and no alternate model was able to review it either. Consider splitting this PR into smaller changes. Tracking id `r-be1e81-81ee69`. Comment `@pr-reviewer-bot retry` to try again.
rcsheets force-pushed perf/nvfp4-gemv-rows from c89e874fa0
All checks were successful
ci / test_and_build (pull_request) Successful in 27s
to ff5944a07c
All checks were successful
ci / test_and_build (pull_request) Successful in 32s
2026-10-01 08:49:04 +00:00
Compare
rcsheets deleted branch perf/nvfp4-gemv-rows 2026-10-01 08:59:10 +00:00
Sign in to join this conversation.
No reviewers
No labels
No milestone
No project
No assignees
2 participants
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
brooktrails/gllm!102
No description provided.