feat: soft and hard thinking-token budgets for reasoning turns #110
Loading…
Reference in a new issue
No description provided.
Delete branch "feat/thinking-budgets"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Reasoning models such as Nemotron 3 Nano can think for thousands of tokens before answering, and a client had no way to bound that short of
max_tokens, which ends the turn with no answer at all. This adds two per-request caps on the reasoning block of a chat turn.API
/v1/chat/completionsaccepts:max_thinking_tokens_hard— the reasoning block never runs past this many tokens; it is closed immediately when it reaches the budget.max_thinking_tokens_soft— once the open reasoning block has generated this many tokens, it is closed at the next sentence boundary.Both are optional and independent; each must be >= 0, and soft must not exceed hard (400 otherwise). 0 is a real budget: a hard budget of 0 closes the block before any reasoning. Reasoning tokens still count against
max_tokens. A turn that does not reason (thinking off, or a template with no reasoning mode) ignores both, since a cap on reasoning is trivially met.How the block is closed
When a budget fires,
sampleBatchemits the chat template's own closing sequence,\n</think>\n, in place of sampled tokens, one token a step (closeReasoning, queued onSequence.Forced). The marker then goes through the same path as one the model sampled, so the reasoning/answer split, streaming deltas,reasoning_tokensusage, stop strings andresponse_formatgrammars all behave as for a block the model closed itself.response_format, since every token past the marker belongs to the grammar and a forced one would bypass its mask."\n"gets the bare marker.reasoning_tokensnever exceeds it.Forcing the whole sequence rather than a bare
</think>matters. Cut mid-thought with a bare marker, Nemotron 3 Nano treated it as noise and carried on reasoning in its answer in 11 of 22 cuts — in one case by emitting a<tool_call>to a Python executor the request never offered — and the leaked reasoning made those turns longer. With the template's sequence it answered cleanly in all 22 (table below).The soft cut is decided on the token after a sentence ends (
sentenceBoundary): the closing sequence replaces the token that follows a line break, one that opens with a newline, or one that opens with whitespace when the text so far ends in.,!or?(closing quotes and brackets aside). The line-break case is the common one on byte-level vocabularies, which fold the break into the punctuation (Nemotron 3 tokenizes.\n\n,?\nand.\nas single tokens). Deciding on the following token keeps "3.14" whole, and a line holding only a list number ("2.") does not count. An abbreviation ("e.g. ") does, which at worst cuts one sentence early. If the hard budget is reached before a boundary, the hard cut wins.The
engine: request completelog line records the budgets that were set andreasoning_cut=soft|hardwhen one of them closed the block.glchat
/thinkingsets reasoning and its budgets for every turn from the next one on:on,offanddefaultsetenable_thinking(defaultleaves it to the chat template, as without--think);soft=Nandhard=Nset the two budgets, and=offclears one. Words left out keep their value, a line with any bad word is refused whole (the input stays for correcting), soft above hard is refused client-side before it can fail a turn, and bare/thinkingreports the setting. Budgets stay off the wire until set, so a server without this PR sees the request glchat always sent.--thinkis deprecated in favor of/thinking: it still works, but is hidden from--helpand prints a deprecation notice when used. It can be removed in a later release.Choosing budgets
A follow-up measurement on Nemotron 3 Nano (12 problems with checkable answers × 4 seeds at temperature 1.0, hard budgets of 64/128/256 each alone or with a soft budget 16, 32 or 64 tokens below; 515 budget-cut turns) found:
A leak here is an answer containing a stray
</think>or<tool_call>: the model kept reasoning in the answer and closed the block itself. That check matches hand labels exactly on the bare-marker runs (11/22) and finds none in the 284 greedy cut turns with the template sequence. Past the soft budget the model reached a sentence boundary a median 11 tokens later (90% within 27). Accuracy falls at tight budgets: 48/48 unbudgeted, 29-38 at 64-128, 37-41 at 256. The README's new Thinking budgets section turns this into guidance: pair every hard budget with a soft one, leave 32-64 tokens between them, and keep the hard budget around 256 or more for multi-step problems.Testing
Engine tests run a deterministic turn plainly, then with budgets, and check the model's own tokens, then the exact closing sequence, then the answer, with reasoning, answer and streamed deltas partitioning the output. Cases cover hard budgets of 0, 1 and 4 (never exceeded), the bare-marker fallback for a tokenizer with no newline, a soft cut at a sentence boundary, a hard budget beating a pending soft one, budgets the turn never reaches, a turn with thinking off, and a cut under
response_format(no trailing newline; the answer is grammatical from its first byte).TestSentenceBoundarypins the boundary rule case by case.glchat tests cover
/thinkingsetting, keeping and clearing values, bare/thinkingchanging nothing, bad lines refused whole, and the budgets reaching the request body only when set.Server tests cover passing both fields through (0 included) and rejecting negative values and soft > hard.
go vet ./...andgo test ./...pass.Served NVIDIA-Nemotron-3-Nano-30B-A3B-NVFP4 on the RTX PRO 6000, greedy, 6 reasoning prompts (word problems, counting, ordering, code) × hard budgets of 40, 70, 100 and 150; 22 of the 24 runs were cut (in the other two the model finished thinking within the budget):
reasoning_tokensover budget</think>\n</think>\n(this PR)Tight budgets still cost accuracy: some cut turns answered wrongly under either sequence, which no closing sequence can fix.
Also on the GPU: soft 64 on a 612-token reasoning turn cut after "= 6:10 pm.\n\n" at 84 tokens; a streamed request split cleanly into reasoning and content deltas;
response_format: json_objectwith a hard budget of 20 returned 20 reasoning tokens and valid JSON; negative budgets and soft > hard returned 400.🤖 Generated with Claude Code
Automated review by pr-reviewer v0.54.0 | Safety Check | Nemotron 3 Nano | tracking id
r-c0ab64-8e1fe1This is an AI-generated review and may contain mistakes.
Status: ❌ Failed
Review failed. Tracking id
r-c0ab64-8e1fe1— see logs for details.Comment
@pr-reviewer-bot retryto try again.glchat could only choose whether a model reasons, and only at startup (--think). /thinking changes it mid-conversation and sets the server's reasoning budgets, for every turn from the next one on: /thinking on soft=512 hard=1024 /thinking default hard=off on, off and default set enable_thinking (default leaves it to the template, as without --think); soft=N and hard=N set max_thinking_tokens_soft and max_thinking_tokens_hard, and =off clears one. Words left out keep their value, a line with any bad word is refused whole, soft above hard is refused before it can fail a turn, and bare /thinking reports the setting. Budgets stay off the wire until set. The help and completion lists pad usage to a fixed column; a usage longer than it now runs over with a gap instead of running into its summary. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>feat(engine): soft and hard thinking-token budgets for reasoning turnsto feat: soft and hard thinking-token budgets for reasoning turns5e6087d4bc727afbe575