perf(engine): read logits into a reused buffer and pick greedy tokens in parallel #105
Loading…
Reference in a new issue
No description provided.
Delete branch "perf/host-sampling"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Problem
After #102, a profiled Nemotron 3 Nano decode step (8 concurrent requests, 19.4 ms under the profiler) leaves the GPU idle for 1.9 ms between one step's logits readback and the next step's first kernel. Most of that is the engine handling the logits on the host:
sampleBatchallocated a byte buffer for the logits, copied the device tensor into it, allocated a[]float32, and converted one into the other element by element. For 8 rows of a 131072-entry vocabulary that is 4 MB each way and about 1 ms.Both costs grow with the number of sequences in the step.
Change
readLogitscopies the device bytes straight into a[]float32that the step loop keeps and reuses. The rows are only read while that step is sampled, and nothing retains them (logprobs.Reducecopies what it reports; the grammar mask edits the row in place, as before). The byte view assumes a little-endian host, as the CPU backend's views already do.greedyTokensfinds the argmax of every greedy, grammar-free row in parallel before the per-sequence loop. Rows that sample with a temperature or carry a grammar are still sampled in the loop, in order, each with its own RNG.The tokens chosen are the same in every case.
Measurements
RTX PRO 6000 Blackwell, 300 W cap, NVIDIA-Nemotron-3-Nano-30B-A3B-NVFP4 with
--kv-cache 8GiB --max-model-len 8192, 8 concurrent chat requests × 256 tokens, greedy, measured after one warmup pass:Verification
TestGreedyTokenscovers the row selection: greedy rows get their argmax with ties to the lower index; sampled, grammar-constrained and finished rows are left to the loop.go test ./...passes, andgo test -race ./internal/engine/passes.Not measured: workloads that sample with a temperature (only the buffer reuse applies to them) and other concurrency levels.
🤖 Generated with Claude Code
Automated review by pr-reviewer v0.52.3 | Safety Check | Nemotron 3 Nano | tracking id
r-bf5a2f-95c035This is an AI-generated review and may contain mistakes.
Status: ❌ Failed
Review failed. Tracking id
r-bf5a2f-95c035— see logs for details.Comment
@pr-reviewer-bot retryto try again.