perf(engine): sample without sorting the vocabulary, rows side by side #112
Loading…
Reference in a new issue
No description provided.
Delete branch "perf/sampling"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Problem
sampleToken(internal/engine/sample.go) built a candidate slice of the whole vocabulary andsort.Sliced it by logit for every sampled (non-greedy) token, before looking at top-k or top-p, so the sort ran even with no truncation at all.Engine.sampleBatchthen called it once per sequence, one after another, under the engine lock. At a 131072-token vocabulary that is ~18 ms a row, so a decode step at 16 sampled sequences spent ~285 ms in sampling alone. Greedy decoding, which takes the parallelgreedyTokensargmax path, was several times faster end to end (a Granite 4.2 run at temperature 1.0 / top_p 0.95 completed about 5 requests a minute at concurrency 16).Change
Sampling without a full sort
The draw walks candidates from the most likely down, subtracting each weight from a uniform point in the total; which token a seed picks depends on that order, but the order only matters where the walk stops.
sampleTokennow uses aranker:(logit - max) / temperature, a quarter nat to a bin (1024 bins down to e^-256, plus one for everything lower, -Inf and NaN), and sums each bin's weight. Every token in a bin outranks every token in a later bin.top_k0,top_p<= 0 or >= 1): the draw skips whole bins by their summed mass and sorts only the bin it lands in.top_p; only that bin is sorted to find the exact cut. The draw over the nucleus then works as above.drawSorted) runs on those k candidates unchanged.Collecting a bin's members screens the row with a float32 bracket around the bin's logit range before the exact bin test, which keeps those rescans cheap. What remains is one O(V) pass of
math.Exp, needed for the softmax total.The previous algorithm is kept as
sampleSorted. It is the reference the tests compare against, and the fallback for rows the bins do not handle: no finite maximum, or a negative temperature.Semantics are unchanged. Temperature, top-k, and top-p mean what they did, including the edges:
top_p<= 0 or >= 1 means no nucleus cut, top-k >= vocab means no cut, and a cumulative probability that round-off keeps short oftop_pkeeps everything. The weights are computed by the same expression as before. Seeded output is preserved: the walk visits candidates in the same order, so a seed draws the same token as before except where float round-off in the per-bin sums moves a draw across a boundary. Run against a verbatim copy of the old function on the benchmark rows (131072 tokens, both shapes, all 13 settings insampleCases, 200 seeded draws each), no draw differed. One deliberate difference: tied logits now go to the lower token id, wheresort.Slice(not stable) left their order unspecified. No test or golden pins sampled output. The seeded engine tests check reproducibility run to run, not specific tokens.Rows sampled side by side
sampleBatchnow runs in three phases:Forcedqueue or the hard thinking budget), and fetches each sampling sequence's RNG fromrngFor.sampleTokenssamples the rows that need a draw in parallel, the same waygreedyTokensdoes argmax.Grammar.Accept, logprobs, decode, and finish rules.A draw reads only its own row, RNG, and parameters, and each phase touches only its own sequence, so the reordering changes no outcome.
sampleTokenstill never writes torow, whichlogprobs.Reducereads afterward.Verification
CPU benchmarks (
go test ./internal/engine -bench BenchmarkSampleToken, AMD Ryzen Threadripper PRO 9955WX, median of 5). The "peaked" rows have a few candidates ~20 above a Gaussian bulk (small nucleus). The "flat" rows have them ~14 above it, so the bulk holds most of the mass (large nucleus, the costly case for top-p). All settings use temperature 1.0.One decode step's sampling, 16 sequences at 131072 tokens with top_p 0.95, half peaked and half flat (
BenchmarkSampleTokens): ~285 ms before (the old function run sequentially) -> ~2.3 ms after. Allocation per sampled row drops from 2-3 MB to under 30 KB.New tests in
sample_test.go:TestSampleMatchesSorted: the binned path draws the same token as the full-sort reference, seed for seed. It covers rows with a Gaussian bulk at three spreads, heavy ties, and grammar-masked -Inf entries, under 13 settings: temperatures 0.02 to 40,top_p0 / 0.5 / 0.9 / 0.95 / 0.999 / 1, andtop_k1 / 40 / 5000, alone and combined.TestSampleStaysInTruncation: top-k, top-p, and both never return a token outside the allowed set, computed independently.TestSampleFrequencies: over 100000 seeded draws, frequencies match the softmax within 4 standard errors, with and without a top-p cut (renormalized over the nucleus) and at a non-unit temperature.TestSampleEdges: a row masked down to one token returns it under every setting; rows with no finite maximum and a negative temperature take the fallback without panicking; the logits row is never written.TestSampleTokens: the parallel batch helper fills exactly the rows given an RNG, each with the token its RNG draws.A deliberate perturbation of the bin-skip test is caught by
TestSampleMatchesSorted.go build ./...,go vet ./...,go test ./..., andgo test -race ./internal/engine/pass, and gofmt is clean. Everything here is CPU-side and was measured on the CPU only; no end-to-end GPU serving run is part of this change.🤖 Generated with Claude Code
Automated review by pr-reviewer v0.54.0 | Safety Check | Nemotron 3 Nano | tracking id
r-c156ad-3456d7This is an AI-generated review and may contain mistakes.
Status: ❌ Failed
Review failed. Tracking id
r-c156ad-3456d7— see logs for details.Comment
@pr-reviewer-bot retryto try again.