perf(engine): read logits into a reused buffer and pick greedy tokens in parallel #105

Merged
rcsheets merged 1 commit from perf/host-sampling into main 2026-10-02 07:28:41 +00:00
Owner

Problem

After #102, a profiled Nemotron 3 Nano decode step (8 concurrent requests, 19.4 ms under the profiler) leaves the GPU idle for 1.9 ms between one step's logits readback and the next step's first kernel. Most of that is the engine handling the logits on the host:

  • sampleBatch allocated a byte buffer for the logits, copied the device tensor into it, allocated a []float32, and converted one into the other element by element. For 8 rows of a 131072-entry vocabulary that is 4 MB each way and about 1 ms.
  • Each greedy row was then scanned for its largest logit one after another, about 50 µs a row.

Both costs grow with the number of sequences in the step.

Change

  • readLogits copies the device bytes straight into a []float32 that the step loop keeps and reuses. The rows are only read while that step is sampled, and nothing retains them (logprobs.Reduce copies what it reports; the grammar mask edits the row in place, as before). The byte view assumes a little-endian host, as the CPU backend's views already do.
  • greedyTokens finds the argmax of every greedy, grammar-free row in parallel before the per-sequence loop. Rows that sample with a temperature or carry a grammar are still sampled in the loop, in order, each with its own RNG.

The tokens chosen are the same in every case.

Measurements

RTX PRO 6000 Blackwell, 300 W cap, NVIDIA-Nemotron-3-Nano-30B-A3B-NVFP4 with --kv-cache 8GiB --max-model-len 8192, 8 concurrent chat requests × 256 tokens, greedy, measured after one warmup pass:

main this branch
output tokens per second 448.2 469.8
GPU energy per token 0.41 J 0.41 J

Verification

  • Eight sequential greedy requests (four ~5k-token prefills at 64 tokens, four short chats at 128) return byte-identical replies on main and on this branch.
  • TestGreedyTokens covers the row selection: greedy rows get their argmax with ties to the lower index; sampled, grammar-constrained and finished rows are left to the loop.
  • go test ./... passes, and go test -race ./internal/engine/ passes.
  • "What is 17*23? Answer briefly." returns 391.

Not measured: workloads that sample with a temperature (only the buffer reuse applies to them) and other concurrency levels.

🤖 Generated with Claude Code

## Problem After #102, a profiled Nemotron 3 Nano decode step (8 concurrent requests, 19.4 ms under the profiler) leaves the GPU idle for 1.9 ms between one step's logits readback and the next step's first kernel. Most of that is the engine handling the logits on the host: - `sampleBatch` allocated a byte buffer for the logits, copied the device tensor into it, allocated a `[]float32`, and converted one into the other element by element. For 8 rows of a 131072-entry vocabulary that is 4 MB each way and about 1 ms. - Each greedy row was then scanned for its largest logit one after another, about 50 µs a row. Both costs grow with the number of sequences in the step. ## Change - `readLogits` copies the device bytes straight into a `[]float32` that the step loop keeps and reuses. The rows are only read while that step is sampled, and nothing retains them (`logprobs.Reduce` copies what it reports; the grammar mask edits the row in place, as before). The byte view assumes a little-endian host, as the CPU backend's views already do. - `greedyTokens` finds the argmax of every greedy, grammar-free row in parallel before the per-sequence loop. Rows that sample with a temperature or carry a grammar are still sampled in the loop, in order, each with its own RNG. The tokens chosen are the same in every case. ## Measurements RTX PRO 6000 Blackwell, 300 W cap, NVIDIA-Nemotron-3-Nano-30B-A3B-NVFP4 with `--kv-cache 8GiB --max-model-len 8192`, 8 concurrent chat requests × 256 tokens, greedy, measured after one warmup pass: | | main | this branch | | --- | ---: | ---: | | output tokens per second | 448.2 | 469.8 | | GPU energy per token | 0.41 J | 0.41 J | ## Verification - Eight sequential greedy requests (four ~5k-token prefills at 64 tokens, four short chats at 128) return byte-identical replies on main and on this branch. - `TestGreedyTokens` covers the row selection: greedy rows get their argmax with ties to the lower index; sampled, grammar-constrained and finished rows are left to the loop. - `go test ./...` passes, and `go test -race ./internal/engine/` passes. - "What is 17*23? Answer briefly." returns 391. Not measured: workloads that sample with a temperature (only the buffer reuse applies to them) and other concurrency levels. 🤖 Generated with [Claude Code](https://claude.com/claude-code)
perf(engine): read logits into a reused buffer and pick greedy tokens in parallel
All checks were successful
ci / test_and_build (pull_request) Successful in 36s
8461a1f26f
After #102 a profiled Nemotron 3 Nano decode step (8 requests, 19.4 ms)
left the GPU idle for 1.9 ms between a step's logits readback and the
next step's first kernel. Most of that was the engine handling the
logits on the host:

- sampleBatch allocated a byte buffer for the logits, copied the device
  tensor into it, allocated a float32 slice and converted one into the
  other element by element: about 1 ms for 8 rows of a 131072-entry
  vocabulary.
- Each greedy row was then scanned for its largest logit one after
  another, 50 us a row.

readLogits copies the device bytes straight into a float32 buffer the
step loop keeps and reuses; the rows are only read while that step is
sampled. greedyTokens finds the argmax of every greedy, grammar-free
row side by side before the per-sequence loop, which still samples the
other rows in order with their own RNGs. The tokens chosen are the same.

Measured on the RTX PRO 6000 Blackwell (300 W cap), Nemotron 3 Nano
NVFP4, 8 concurrent requests x 256 tokens: 448 -> 470 tok/s. Eight
sequential greedy requests (four ~5k-token prefills, four short chats)
produced byte-identical replies before and after.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Collaborator

Automated review by pr-reviewer v0.52.3 | Safety Check | Nemotron 3 Nano | tracking id r-bf5a2f-95c035
This is an AI-generated review and may contain mistakes.

Status: ❌ Failed


Review failed. Tracking id r-bf5a2f-95c035 — see logs for details.

Comment @pr-reviewer-bot retry to try again.

<!-- pr-reviewer:review --> *Automated review by [pr-reviewer](https://git.brooktrails.org/brooktrails/pr-reviewer) v0.52.3 | Safety Check | Nemotron 3 Nano | tracking id `r-bf5a2f-95c035`* *This is an AI-generated review and may contain mistakes.* **Status:** ❌ Failed --- Review failed. Tracking id `r-bf5a2f-95c035` — see logs for details. Comment `@pr-reviewer-bot retry` to try again.
rcsheets deleted branch perf/host-sampling 2026-10-02 07:28:45 +00:00
Sign in to join this conversation.
No reviewers
No labels
No milestone
No project
No assignees
2 participants
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
brooktrails/gllm!105
No description provided.