test(cuda): add cold-weight and skewed cases to the grouped GEMV benchmark #106
Loading…
Reference in a new issue
No description provided.
Delete branch "test/gemv-bench-cold"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Problem
BenchmarkCUDANVFP4GEMVGroupedre-reads the same experts on every launch and gives every expert the same number of rows. It therefore measures a launch whose weights sit in the L2 cache and whose groups are uniform. A served Nemotron 3 Nano step reads 4.8 GB of expert weights, and its groups are mostly one row with a few of several. In a profiled 8-request step the grouped launches averaged 153 µs, where the benchmark's nearest case reports 116 µs.Change
Two kinds of case are added, alone and combined:
The existing cases are unchanged. The benchmark now uploads 512 copies of the weight per projection instead of 48, which is 1.4 GB of device memory while it runs.
Measurements
RTX PRO 6000 Blackwell, 300 W cap, up projection (1856×2688), 8 tokens over 37 experts:
Verification
The benchmark ran on the Blackwell with all cases, both projections.
go vetand thecuda-tagged test build are clean. No non-test code changes.🤖 Generated with Claude Code
Automated review by pr-reviewer v0.52.3 | Safety Check | Nemotron 3 Nano | tracking id
r-bf5a8f-b88923This is an AI-generated review and may contain mistakes.
Status: ❌ Failed
Review failed. Tracking id
r-bf5a8f-b88923— see logs for details.Comment
@pr-reviewer-bot retryto try again.