perf(engine): sample without sorting the vocabulary, rows side by side #112

Merged
rcsheets merged 1 commit from perf/sampling into main 2026-10-03 20:18:18 +00:00
Owner

Problem

sampleToken (internal/engine/sample.go) built a candidate slice of the whole vocabulary and sort.Sliced it by logit for every sampled (non-greedy) token, before looking at top-k or top-p, so the sort ran even with no truncation at all. Engine.sampleBatch then called it once per sequence, one after another, under the engine lock. At a 131072-token vocabulary that is ~18 ms a row, so a decode step at 16 sampled sequences spent ~285 ms in sampling alone. Greedy decoding, which takes the parallel greedyTokens argmax path, was several times faster end to end (a Granite 4.2 run at temperature 1.0 / top_p 0.95 completed about 5 requests a minute at concurrency 16).

Change

Sampling without a full sort

The draw walks candidates from the most likely down, subtracting each weight from a uniform point in the total; which token a seed picks depends on that order, but the order only matters where the walk stops. sampleToken now uses a ranker:

  • One pass bins every token by its log-weight (logit - max) / temperature, a quarter nat to a bin (1024 bins down to e^-256, plus one for everything lower, -Inf and NaN), and sums each bin's weight. Every token in a bin outranks every token in a later bin.
  • No truncation (top_k 0, top_p <= 0 or >= 1): the draw skips whole bins by their summed mass and sorts only the bin it lands in.
  • top-p: whole bins are taken into the nucleus by mass until one would reach top_p; only that bin is sorted to find the exact cut. The draw over the nucleus then works as above.
  • top-k: a counting pass finds the bin holding the k-th token; only bins up to it are collected and sorted, and the existing softmax / top-p / draw code (drawSorted) runs on those k candidates unchanged.

Collecting a bin's members screens the row with a float32 bracket around the bin's logit range before the exact bin test, which keeps those rescans cheap. What remains is one O(V) pass of math.Exp, needed for the softmax total.

The previous algorithm is kept as sampleSorted. It is the reference the tests compare against, and the fallback for rows the bins do not handle: no finite maximum, or a negative temperature.

Semantics are unchanged. Temperature, top-k, and top-p mean what they did, including the edges: top_p <= 0 or >= 1 means no nucleus cut, top-k >= vocab means no cut, and a cumulative probability that round-off keeps short of top_p keeps everything. The weights are computed by the same expression as before. Seeded output is preserved: the walk visits candidates in the same order, so a seed draws the same token as before except where float round-off in the per-bin sums moves a draw across a boundary. Run against a verbatim copy of the old function on the benchmark rows (131072 tokens, both shapes, all 13 settings in sampleCases, 200 seeded draws each), no draw differed. One deliberate difference: tied logits now go to the lower token id, where sort.Slice (not stable) left their order unspecified. No test or golden pins sampled output. The seeded engine tests check reproducibility run to run, not specific tokens.

Rows sampled side by side

sampleBatch now runs in three phases:

  1. Under the lock and in order, it applies grammar and pending-grammar EOS masks, settles forced tokens (an existing Forced queue or the hard thinking budget), and fetches each sampling sequence's RNG from rngFor.
  2. sampleTokens samples the rows that need a draw in parallel, the same way greedyTokens does argmax.
  3. The existing per-sequence loop does accounting, the soft-budget sentence-boundary check (on sequences that picked their own token), forced-token substitution, Grammar.Accept, logprobs, decode, and finish rules.

A draw reads only its own row, RNG, and parameters, and each phase touches only its own sequence, so the reordering changes no outcome. sampleToken still never writes to row, which logprobs.Reduce reads afterward.

Verification

CPU benchmarks (go test ./internal/engine -bench BenchmarkSampleToken, AMD Ryzen Threadripper PRO 9955WX, median of 5). The "peaked" rows have a few candidates ~20 above a Gaussian bulk (small nucleus). The "flat" rows have them ~14 above it, so the bulk holds most of the mass (large nucleus, the costly case for top-p). All settings use temperature 1.0.

row before after speedup
100352 peaked, top_p=1 13.75 ms 0.775 ms 18x
100352 peaked, top_p=0.95 13.65 ms 0.811 ms 17x
100352 peaked, top_k=50 12.95 ms 0.149 ms 87x
100352 flat, top_p=1 13.61 ms 0.762 ms 18x
100352 flat, top_p=0.95 13.66 ms 0.983 ms 14x
100352 flat, top_k=50 12.84 ms 0.145 ms 88x
131072 peaked, top_p=1 18.06 ms 1.010 ms 18x
131072 peaked, top_p=0.95 18.04 ms 1.069 ms 17x
131072 peaked, top_k=50 17.16 ms 0.186 ms 92x
131072 flat, top_p=1 18.27 ms 0.999 ms 18x
131072 flat, top_p=0.95 18.06 ms 1.372 ms 13x
131072 flat, top_k=50 17.22 ms 0.183 ms 94x

One decode step's sampling, 16 sequences at 131072 tokens with top_p 0.95, half peaked and half flat (BenchmarkSampleTokens): ~285 ms before (the old function run sequentially) -> ~2.3 ms after. Allocation per sampled row drops from 2-3 MB to under 30 KB.

New tests in sample_test.go:

  • TestSampleMatchesSorted: the binned path draws the same token as the full-sort reference, seed for seed. It covers rows with a Gaussian bulk at three spreads, heavy ties, and grammar-masked -Inf entries, under 13 settings: temperatures 0.02 to 40, top_p 0 / 0.5 / 0.9 / 0.95 / 0.999 / 1, and top_k 1 / 40 / 5000, alone and combined.
  • TestSampleStaysInTruncation: top-k, top-p, and both never return a token outside the allowed set, computed independently.
  • TestSampleFrequencies: over 100000 seeded draws, frequencies match the softmax within 4 standard errors, with and without a top-p cut (renormalized over the nucleus) and at a non-unit temperature.
  • TestSampleEdges: a row masked down to one token returns it under every setting; rows with no finite maximum and a negative temperature take the fallback without panicking; the logits row is never written.
  • TestSampleTokens: the parallel batch helper fills exactly the rows given an RNG, each with the token its RNG draws.

A deliberate perturbation of the bin-skip test is caught by TestSampleMatchesSorted. go build ./..., go vet ./..., go test ./..., and go test -race ./internal/engine/ pass, and gofmt is clean. Everything here is CPU-side and was measured on the CPU only; no end-to-end GPU serving run is part of this change.

🤖 Generated with Claude Code

## Problem `sampleToken` (`internal/engine/sample.go`) built a candidate slice of the whole vocabulary and `sort.Slice`d it by logit for every sampled (non-greedy) token, before looking at top-k or top-p, so the sort ran even with no truncation at all. `Engine.sampleBatch` then called it once per sequence, one after another, under the engine lock. At a 131072-token vocabulary that is ~18 ms a row, so a decode step at 16 sampled sequences spent ~285 ms in sampling alone. Greedy decoding, which takes the parallel `greedyTokens` argmax path, was several times faster end to end (a Granite 4.2 run at temperature 1.0 / top_p 0.95 completed about 5 requests a minute at concurrency 16). ## Change ### Sampling without a full sort The draw walks candidates from the most likely down, subtracting each weight from a uniform point in the total; which token a seed picks depends on that order, but the order only matters where the walk stops. `sampleToken` now uses a `ranker`: - One pass bins every token by its log-weight `(logit - max) / temperature`, a quarter nat to a bin (1024 bins down to e^-256, plus one for everything lower, -Inf and NaN), and sums each bin's weight. Every token in a bin outranks every token in a later bin. - **No truncation** (`top_k` 0, `top_p` <= 0 or >= 1): the draw skips whole bins by their summed mass and sorts only the bin it lands in. - **top-p**: whole bins are taken into the nucleus by mass until one would reach `top_p`; only that bin is sorted to find the exact cut. The draw over the nucleus then works as above. - **top-k**: a counting pass finds the bin holding the k-th token; only bins up to it are collected and sorted, and the existing softmax / top-p / draw code (`drawSorted`) runs on those k candidates unchanged. Collecting a bin's members screens the row with a float32 bracket around the bin's logit range before the exact bin test, which keeps those rescans cheap. What remains is one O(V) pass of `math.Exp`, needed for the softmax total. The previous algorithm is kept as `sampleSorted`. It is the reference the tests compare against, and the fallback for rows the bins do not handle: no finite maximum, or a negative temperature. **Semantics are unchanged.** Temperature, top-k, and top-p mean what they did, including the edges: `top_p` <= 0 or >= 1 means no nucleus cut, top-k >= vocab means no cut, and a cumulative probability that round-off keeps short of `top_p` keeps everything. The weights are computed by the same expression as before. **Seeded output is preserved**: the walk visits candidates in the same order, so a seed draws the same token as before except where float round-off in the per-bin sums moves a draw across a boundary. Run against a verbatim copy of the old function on the benchmark rows (131072 tokens, both shapes, all 13 settings in `sampleCases`, 200 seeded draws each), no draw differed. One deliberate difference: tied logits now go to the lower token id, where `sort.Slice` (not stable) left their order unspecified. No test or golden pins sampled output. The seeded engine tests check reproducibility run to run, not specific tokens. ### Rows sampled side by side `sampleBatch` now runs in three phases: 1. Under the lock and in order, it applies grammar and pending-grammar EOS masks, settles forced tokens (an existing `Forced` queue or the hard thinking budget), and fetches each sampling sequence's RNG from `rngFor`. 2. `sampleTokens` samples the rows that need a draw in parallel, the same way `greedyTokens` does argmax. 3. The existing per-sequence loop does accounting, the soft-budget sentence-boundary check (on sequences that picked their own token), forced-token substitution, `Grammar.Accept`, logprobs, decode, and finish rules. A draw reads only its own row, RNG, and parameters, and each phase touches only its own sequence, so the reordering changes no outcome. `sampleToken` still never writes to `row`, which `logprobs.Reduce` reads afterward. ## Verification CPU benchmarks (`go test ./internal/engine -bench BenchmarkSampleToken`, AMD Ryzen Threadripper PRO 9955WX, median of 5). The "peaked" rows have a few candidates ~20 above a Gaussian bulk (small nucleus). The "flat" rows have them ~14 above it, so the bulk holds most of the mass (large nucleus, the costly case for top-p). All settings use temperature 1.0. | row | before | after | speedup | |---|---|---|---| | 100352 peaked, top_p=1 | 13.75 ms | 0.775 ms | 18x | | 100352 peaked, top_p=0.95 | 13.65 ms | 0.811 ms | 17x | | 100352 peaked, top_k=50 | 12.95 ms | 0.149 ms | 87x | | 100352 flat, top_p=1 | 13.61 ms | 0.762 ms | 18x | | 100352 flat, top_p=0.95 | 13.66 ms | 0.983 ms | 14x | | 100352 flat, top_k=50 | 12.84 ms | 0.145 ms | 88x | | 131072 peaked, top_p=1 | 18.06 ms | 1.010 ms | 18x | | 131072 peaked, top_p=0.95 | 18.04 ms | 1.069 ms | 17x | | 131072 peaked, top_k=50 | 17.16 ms | 0.186 ms | 92x | | 131072 flat, top_p=1 | 18.27 ms | 0.999 ms | 18x | | 131072 flat, top_p=0.95 | 18.06 ms | 1.372 ms | 13x | | 131072 flat, top_k=50 | 17.22 ms | 0.183 ms | 94x | One decode step's sampling, 16 sequences at 131072 tokens with top_p 0.95, half peaked and half flat (`BenchmarkSampleTokens`): **~285 ms before** (the old function run sequentially) **-> ~2.3 ms after**. Allocation per sampled row drops from 2-3 MB to under 30 KB. New tests in `sample_test.go`: - `TestSampleMatchesSorted`: the binned path draws the same token as the full-sort reference, seed for seed. It covers rows with a Gaussian bulk at three spreads, heavy ties, and grammar-masked -Inf entries, under 13 settings: temperatures 0.02 to 40, `top_p` 0 / 0.5 / 0.9 / 0.95 / 0.999 / 1, and `top_k` 1 / 40 / 5000, alone and combined. - `TestSampleStaysInTruncation`: top-k, top-p, and both never return a token outside the allowed set, computed independently. - `TestSampleFrequencies`: over 100000 seeded draws, frequencies match the softmax within 4 standard errors, with and without a top-p cut (renormalized over the nucleus) and at a non-unit temperature. - `TestSampleEdges`: a row masked down to one token returns it under every setting; rows with no finite maximum and a negative temperature take the fallback without panicking; the logits row is never written. - `TestSampleTokens`: the parallel batch helper fills exactly the rows given an RNG, each with the token its RNG draws. A deliberate perturbation of the bin-skip test is caught by `TestSampleMatchesSorted`. `go build ./...`, `go vet ./...`, `go test ./...`, and `go test -race ./internal/engine/` pass, and gofmt is clean. Everything here is CPU-side and was measured on the CPU only; no end-to-end GPU serving run is part of this change. 🤖 Generated with [Claude Code](https://claude.com/claude-code)
perf(engine): sample without sorting the vocabulary, rows side by side
All checks were successful
ci / test_and_build (pull_request) Successful in 55s
93ea6e9f44
sampleToken sorted the whole vocabulary by logit for every sampled token,
whatever top-k/top-p asked for: ~18 ms a row at 131072 tokens, run one
sequence after another under the engine lock, so a sampled decode step at
16 sequences spent ~285 ms on sampling alone.

Tokens are now binned by log-weight (a quarter nat to a bin) in one pass
that also sums each bin's mass. The draw still walks candidates from the
most likely down, but skips whole bins by their mass and sorts only the bin
where the nucleus ends and the bin where the draw lands; top-k counts bins
to find the k-th token and sorts only that prefix. The walk is the full
sort's, so a seed draws the same token as before up to float round-off in
the summed masses (no difference seen in 5200 draws on 131072-token rows);
tied logits now go to the lower id instead of sort.Slice's arbitrary order.
The full sort stays as sampleSorted, the reference the tests compare against
and the fallback for rows with no finite maximum or a negative temperature.

sampleBatch now masks rows and settles forced tokens for every sequence
first, then samples the rows that need a draw in parallel (sampleTokens,
like greedyTokens), with RNGs fetched under the lock beforehand.

Per row: 13-18 ms -> 0.15-1.4 ms; 16 rows at 131072: 285 ms -> 2.3 ms.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Collaborator

Automated review by pr-reviewer v0.54.0 | Safety Check | Nemotron 3 Nano | tracking id r-c156ad-3456d7
This is an AI-generated review and may contain mistakes.

Status: ❌ Failed


Review failed. Tracking id r-c156ad-3456d7 — see logs for details.

Comment @pr-reviewer-bot retry to try again.

<!-- pr-reviewer:review --> *Automated review by [pr-reviewer](https://git.brooktrails.org/brooktrails/pr-reviewer) v0.54.0 | Safety Check | Nemotron 3 Nano | tracking id `r-c156ad-3456d7`* *This is an AI-generated review and may contain mistakes.* **Status:** ❌ Failed --- Review failed. Tracking id `r-c156ad-3456d7` — see logs for details. Comment `@pr-reviewer-bot retry` to try again.
rcsheets deleted branch perf/sampling 2026-10-03 20:18:18 +00:00
Sign in to join this conversation.
No reviewers
No labels
No milestone
No project
No assignees
2 participants
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
brooktrails/gllm!112
No description provided.