test(cuda): add cold-weight and skewed cases to the grouped GEMV benchmark #106

Merged
rcsheets merged 1 commit from test/gemv-bench-cold into main 2026-10-02 08:12:13 +00:00
Owner

Problem

BenchmarkCUDANVFP4GEMVGrouped re-reads the same experts on every launch and gives every expert the same number of rows. It therefore measures a launch whose weights sit in the L2 cache and whose groups are uniform. A served Nemotron 3 Nano step reads 4.8 GB of expert weights, and its groups are mostly one row with a few of several. In a profiled 8-request step the grouped launches averaged 153 µs, where the benchmark's nearest case reports 116 µs.

Change

Two kinds of case are added, alone and combined:

  • cold: each launch takes its experts from a different part of a 1.4 GB pool of weights, so they come from memory.
  • skewed: one expert gets 8 rows and four more get 2 each; the rest get 1.

The existing cases are unchanged. The benchmark now uploads 512 copies of the weight per projection instead of 48, which is 1.4 GB of device memory while it runs.

Measurements

RTX PRO 6000 Blackwell, 300 W cap, up projection (1856×2688), 8 tokens over 37 experts:

case launch
even rows, same experts (existing) 116 µs
cold 162 µs
skewed 167 µs
skewed and cold 206 µs

Verification

The benchmark ran on the Blackwell with all cases, both projections. go vet and the cuda-tagged test build are clean. No non-test code changes.

🤖 Generated with Claude Code

## Problem `BenchmarkCUDANVFP4GEMVGrouped` re-reads the same experts on every launch and gives every expert the same number of rows. It therefore measures a launch whose weights sit in the L2 cache and whose groups are uniform. A served Nemotron 3 Nano step reads 4.8 GB of expert weights, and its groups are mostly one row with a few of several. In a profiled 8-request step the grouped launches averaged 153 µs, where the benchmark's nearest case reports 116 µs. ## Change Two kinds of case are added, alone and combined: - **cold**: each launch takes its experts from a different part of a 1.4 GB pool of weights, so they come from memory. - **skewed**: one expert gets 8 rows and four more get 2 each; the rest get 1. The existing cases are unchanged. The benchmark now uploads 512 copies of the weight per projection instead of 48, which is 1.4 GB of device memory while it runs. ## Measurements RTX PRO 6000 Blackwell, 300 W cap, up projection (1856×2688), 8 tokens over 37 experts: | case | launch | | --- | ---: | | even rows, same experts (existing) | 116 µs | | cold | 162 µs | | skewed | 167 µs | | skewed and cold | 206 µs | ## Verification The benchmark ran on the Blackwell with all cases, both projections. `go vet` and the `cuda`-tagged test build are clean. No non-test code changes. 🤖 Generated with [Claude Code](https://claude.com/claude-code)
test(cuda): add cold-weight and skewed cases to the grouped GEMV benchmark
All checks were successful
ci / test_and_build (pull_request) Successful in 27s
5f4d0d24f2
BenchmarkCUDANVFP4GEMVGrouped re-read the same experts on every launch
and gave every expert the same number of rows, so it measured a launch
whose weights sit in the L2 cache and whose groups are uniform. A
served Nemotron 3 Nano step reads 4.8 GB of expert weights and its
groups are mostly one row with a few of several; a profiled step's
launches took 153 us where the benchmark's nearest case took 116.

Two kinds of case close the gap:

- "cold" takes each launch's experts from a different part of a 1.4 GB
  pool of weights, so they come from memory.
- "skewed" gives one expert 8 rows and four more 2 each.

On the RTX PRO 6000 Blackwell (300 W cap), 8 tokens over 37 experts, up
projection: 116 us as before, 162 us cold, 167 us skewed, 206 us both.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Collaborator

Automated review by pr-reviewer v0.52.3 | Safety Check | Nemotron 3 Nano | tracking id r-bf5a8f-b88923
This is an AI-generated review and may contain mistakes.

Status: ❌ Failed


Review failed. Tracking id r-bf5a8f-b88923 — see logs for details.

Comment @pr-reviewer-bot retry to try again.

<!-- pr-reviewer:review --> *Automated review by [pr-reviewer](https://git.brooktrails.org/brooktrails/pr-reviewer) v0.52.3 | Safety Check | Nemotron 3 Nano | tracking id `r-bf5a8f-b88923`* *This is an AI-generated review and may contain mistakes.* **Status:** ❌ Failed --- Review failed. Tracking id `r-bf5a8f-b88923` — see logs for details. Comment `@pr-reviewer-bot retry` to try again.
rcsheets deleted branch test/gemv-bench-cold 2026-10-02 08:12:14 +00:00
Sign in to join this conversation.
No reviewers
No labels
No milestone
No project
No assignees
2 participants
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
brooktrails/gllm!106
No description provided.