perf(cuda): compute several output rows per warp in the NVFP4 GEMV #102
Loading…
Reference in a new issue
No description provided.
Delete branch "perf/nvfp4-gemv-rows"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Problem
A profiled Nemotron 3 Nano decode step (8 concurrent requests, main at
bf7b812) takes 23 ms, and the GPU is busy for 86% of it. The two wide-load NVFP4 GEMV kernels account for 70% of the step: the grouped MoE launch 8.5 ms and the single-matrix launch 7.6 ms. The per-layer MoE gate readbacks, which looked like the next target, are 0.5 ms.A new microbenchmark shows what bounds those kernels. Launch time is linear in the number of products (activation rows × out × in), at about 1.5 × 10¹² products per second for every shape and row count. It does not follow the weight bytes: a kernel that only reads the same weights takes 2.5 µs of a 49 µs 8-row launch. Each warp computed one output row, so every product paid for a quarter of an activation load, and every warp fetched each activation row for itself. Pointing those loads at a small window, with nothing else changed, cut the 8-row launch to 27 µs.
Change
gemv_vec_rows<ROWS, OUTS>replacesgemv_vec_row<ROWS>in both the plain and the grouped kernel.gemv_rows_per_warppicks the largest count that leaves every SM four warps. A single matrix is capped at 2 for one or two rows and at 4 otherwise; a grouped launch has a set of warps per expert and usually takes 8.Measurements
RTX PRO 6000 Blackwell, 300 W cap. Kernel times are from
BenchmarkCUDANVFP4GEMVandBenchmarkCUDANVFP4GEMVGrouped, which re-read the same weights each launch and so serve them from the L2 cache; a served step reads its weights from DRAM.Serving NVIDIA-Nemotron-3-Nano-30B-A3B-NVFP4 with
--kv-cache 8GiB --max-model-len 8192, 8 concurrent chat requests × 256 tokens, greedy, measured after one warmup pass:Verification
nemotronhCUDA tests pass on the Blackwell. The NVFP4 GEMV and grouped tests pass on the RTX 3070.Not run: the
GLLM_CUDA_SYNC=1comparison (execution order is unchanged) and the full glbench matrix against vLLM.Remaining headroom
The single-matrix launch is still short of warps: a lane walks three or four chunks serially. Giving each chunk pass its own warp and combining the partial sums in the same order would triple the warps while keeping results bit-identical. Whether the grouped launch is near the DRAM bandwidth limit in a served step is not yet measured.
🤖 Generated with Claude Code
Automated review by pr-reviewer v0.52.3 | Safety Check | Ministral 3 Instruct | tracking id
r-be1e81-81ee69This is an AI-generated review and may contain mistakes.
Status: ❌ Failed
This review couldn't be completed: that model isn't loaded on the inference service right now, and no alternate model was able to review it either. Consider splitting this PR into smaller changes. Tracking id
r-be1e81-81ee69.Comment
@pr-reviewer-bot retryto try again.c89e874fa0ff5944a07c