feat: soft and hard thinking-token budgets for reasoning turns #110

Merged
rcsheets merged 6 commits from feat/thinking-budgets into main 2026-10-03 18:12:12 +00:00
Owner

Reasoning models such as Nemotron 3 Nano can think for thousands of tokens before answering, and a client had no way to bound that short of max_tokens, which ends the turn with no answer at all. This adds two per-request caps on the reasoning block of a chat turn.

API

/v1/chat/completions accepts:

  • max_thinking_tokens_hard — the reasoning block never runs past this many tokens; it is closed immediately when it reaches the budget.
  • max_thinking_tokens_soft — once the open reasoning block has generated this many tokens, it is closed at the next sentence boundary.

Both are optional and independent; each must be >= 0, and soft must not exceed hard (400 otherwise). 0 is a real budget: a hard budget of 0 closes the block before any reasoning. Reasoning tokens still count against max_tokens. A turn that does not reason (thinking off, or a template with no reasoning mode) ignores both, since a cap on reasoning is trivially met.

How the block is closed

When a budget fires, sampleBatch emits the chat template's own closing sequence, \n</think>\n, in place of sampled tokens, one token a step (closeReasoning, queued on Sequence.Forced). The marker then goes through the same path as one the model sampled, so the reasoning/answer split, streaming deltas, reasoning_tokens usage, stop strings and response_format grammars all behave as for a block the model closed itself.

  • The leading newline is skipped when the text already ends a line (or is empty).
  • The trailing newline is skipped under response_format, since every token past the marker belongs to the grammar and a forced one would bypass its mask.
  • A tokenizer that cannot round-trip "\n" gets the bare marker.
  • The hard budget reserves room for the leading newline, so reasoning_tokens never exceeds it.

Forcing the whole sequence rather than a bare </think> matters. Cut mid-thought with a bare marker, Nemotron 3 Nano treated it as noise and carried on reasoning in its answer in 11 of 22 cuts — in one case by emitting a <tool_call> to a Python executor the request never offered — and the leaked reasoning made those turns longer. With the template's sequence it answered cleanly in all 22 (table below).

The soft cut is decided on the token after a sentence ends (sentenceBoundary): the closing sequence replaces the token that follows a line break, one that opens with a newline, or one that opens with whitespace when the text so far ends in ., ! or ? (closing quotes and brackets aside). The line-break case is the common one on byte-level vocabularies, which fold the break into the punctuation (Nemotron 3 tokenizes .\n\n, ?\n and .\n as single tokens). Deciding on the following token keeps "3.14" whole, and a line holding only a list number ("2.") does not count. An abbreviation ("e.g. ") does, which at worst cuts one sentence early. If the hard budget is reached before a boundary, the hard cut wins.

The engine: request complete log line records the budgets that were set and reasoning_cut=soft|hard when one of them closed the block.

glchat

/thinking sets reasoning and its budgets for every turn from the next one on:

/thinking on soft=512 hard=1024
/thinking default hard=off

on, off and default set enable_thinking (default leaves it to the chat template, as without --think); soft=N and hard=N set the two budgets, and =off clears one. Words left out keep their value, a line with any bad word is refused whole (the input stays for correcting), soft above hard is refused client-side before it can fail a turn, and bare /thinking reports the setting. Budgets stay off the wire until set, so a server without this PR sees the request glchat always sent.

--think is deprecated in favor of /thinking: it still works, but is hidden from --help and prints a deprecation notice when used. It can be removed in a later release.

Choosing budgets

A follow-up measurement on Nemotron 3 Nano (12 problems with checkable answers × 4 seeds at temperature 1.0, hard budgets of 64/128/256 each alone or with a soft budget 16, 32 or 64 tokens below; 515 budget-cut turns) found:

leaks / cut turns
hard budget only, at 64 / 128 / 256 5/48 · 1/47 · 0/30
soft budget below the hard one (any gap) 3/390, none at hard ≥ 128

A leak here is an answer containing a stray </think> or <tool_call>: the model kept reasoning in the answer and closed the block itself. That check matches hand labels exactly on the bare-marker runs (11/22) and finds none in the 284 greedy cut turns with the template sequence. Past the soft budget the model reached a sentence boundary a median 11 tokens later (90% within 27). Accuracy falls at tight budgets: 48/48 unbudgeted, 29-38 at 64-128, 37-41 at 256. The README's new Thinking budgets section turns this into guidance: pair every hard budget with a soft one, leave 32-64 tokens between them, and keep the hard budget around 256 or more for multi-step problems.

Testing

  • Engine tests run a deterministic turn plainly, then with budgets, and check the model's own tokens, then the exact closing sequence, then the answer, with reasoning, answer and streamed deltas partitioning the output. Cases cover hard budgets of 0, 1 and 4 (never exceeded), the bare-marker fallback for a tokenizer with no newline, a soft cut at a sentence boundary, a hard budget beating a pending soft one, budgets the turn never reaches, a turn with thinking off, and a cut under response_format (no trailing newline; the answer is grammatical from its first byte).

  • TestSentenceBoundary pins the boundary rule case by case.

  • glchat tests cover /thinking setting, keeping and clearing values, bare /thinking changing nothing, bad lines refused whole, and the budgets reaching the request body only when set.

  • Server tests cover passing both fields through (0 included) and rejecting negative values and soft > hard.

  • go vet ./... and go test ./... pass.

  • Served NVIDIA-Nemotron-3-Nano-30B-A3B-NVFP4 on the RTX PRO 6000, greedy, 6 reasoning prompts (word problems, counting, ordering, code) × hard budgets of 40, 70, 100 and 150; 22 of the 24 runs were cut (in the other two the model finished thinking within the budget):

    closing sequence answer continues the reasoning answer starts cleanly reasoning_tokens over budget
    bare </think> 11 11 0
    \n</think>\n (this PR) 0 22 0

    Tight budgets still cost accuracy: some cut turns answered wrongly under either sequence, which no closing sequence can fix.

  • Also on the GPU: soft 64 on a 612-token reasoning turn cut after "= 6:10 pm.\n\n" at 84 tokens; a streamed request split cleanly into reasoning and content deltas; response_format: json_object with a hard budget of 20 returned 20 reasoning tokens and valid JSON; negative budgets and soft > hard returned 400.

🤖 Generated with Claude Code

Reasoning models such as Nemotron 3 Nano can think for thousands of tokens before answering, and a client had no way to bound that short of `max_tokens`, which ends the turn with no answer at all. This adds two per-request caps on the reasoning block of a chat turn. ## API `/v1/chat/completions` accepts: - `max_thinking_tokens_hard` — the reasoning block never runs past this many tokens; it is closed immediately when it reaches the budget. - `max_thinking_tokens_soft` — once the open reasoning block has generated this many tokens, it is closed at the next sentence boundary. Both are optional and independent; each must be >= 0, and soft must not exceed hard (400 otherwise). 0 is a real budget: a hard budget of 0 closes the block before any reasoning. Reasoning tokens still count against `max_tokens`. A turn that does not reason (thinking off, or a template with no reasoning mode) ignores both, since a cap on reasoning is trivially met. ## How the block is closed When a budget fires, `sampleBatch` emits the chat template's own closing sequence, `\n</think>\n`, in place of sampled tokens, one token a step (`closeReasoning`, queued on `Sequence.Forced`). The marker then goes through the same path as one the model sampled, so the reasoning/answer split, streaming deltas, `reasoning_tokens` usage, stop strings and `response_format` grammars all behave as for a block the model closed itself. - The leading newline is skipped when the text already ends a line (or is empty). - The trailing newline is skipped under `response_format`, since every token past the marker belongs to the grammar and a forced one would bypass its mask. - A tokenizer that cannot round-trip `"\n"` gets the bare marker. - The hard budget reserves room for the leading newline, so `reasoning_tokens` never exceeds it. Forcing the whole sequence rather than a bare `</think>` matters. Cut mid-thought with a bare marker, Nemotron 3 Nano treated it as noise and carried on reasoning in its answer in 11 of 22 cuts — in one case by emitting a `<tool_call>` to a Python executor the request never offered — and the leaked reasoning made those turns longer. With the template's sequence it answered cleanly in all 22 (table below). The soft cut is decided on the token *after* a sentence ends (`sentenceBoundary`): the closing sequence replaces the token that follows a line break, one that opens with a newline, or one that opens with whitespace when the text so far ends in `.`, `!` or `?` (closing quotes and brackets aside). The line-break case is the common one on byte-level vocabularies, which fold the break into the punctuation (Nemotron 3 tokenizes `.\n\n`, `?\n` and `.\n` as single tokens). Deciding on the following token keeps "3.14" whole, and a line holding only a list number ("2.") does not count. An abbreviation ("e.g. ") does, which at worst cuts one sentence early. If the hard budget is reached before a boundary, the hard cut wins. The `engine: request complete` log line records the budgets that were set and `reasoning_cut=soft|hard` when one of them closed the block. ## glchat `/thinking` sets reasoning and its budgets for every turn from the next one on: /thinking on soft=512 hard=1024 /thinking default hard=off `on`, `off` and `default` set `enable_thinking` (`default` leaves it to the chat template, as without `--think`); `soft=N` and `hard=N` set the two budgets, and `=off` clears one. Words left out keep their value, a line with any bad word is refused whole (the input stays for correcting), soft above hard is refused client-side before it can fail a turn, and bare `/thinking` reports the setting. Budgets stay off the wire until set, so a server without this PR sees the request glchat always sent. `--think` is deprecated in favor of `/thinking`: it still works, but is hidden from `--help` and prints a deprecation notice when used. It can be removed in a later release. ## Choosing budgets A follow-up measurement on Nemotron 3 Nano (12 problems with checkable answers × 4 seeds at temperature 1.0, hard budgets of 64/128/256 each alone or with a soft budget 16, 32 or 64 tokens below; 515 budget-cut turns) found: | | leaks / cut turns | | --- | ---: | | hard budget only, at 64 / 128 / 256 | 5/48 · 1/47 · 0/30 | | soft budget below the hard one (any gap) | 3/390, none at hard ≥ 128 | A leak here is an answer containing a stray `</think>` or `<tool_call>`: the model kept reasoning in the answer and closed the block itself. That check matches hand labels exactly on the bare-marker runs (11/22) and finds none in the 284 greedy cut turns with the template sequence. Past the soft budget the model reached a sentence boundary a median 11 tokens later (90% within 27). Accuracy falls at tight budgets: 48/48 unbudgeted, 29-38 at 64-128, 37-41 at 256. The README's new *Thinking budgets* section turns this into guidance: pair every hard budget with a soft one, leave 32-64 tokens between them, and keep the hard budget around 256 or more for multi-step problems. ## Testing - Engine tests run a deterministic turn plainly, then with budgets, and check the model's own tokens, then the exact closing sequence, then the answer, with reasoning, answer and streamed deltas partitioning the output. Cases cover hard budgets of 0, 1 and 4 (never exceeded), the bare-marker fallback for a tokenizer with no newline, a soft cut at a sentence boundary, a hard budget beating a pending soft one, budgets the turn never reaches, a turn with thinking off, and a cut under `response_format` (no trailing newline; the answer is grammatical from its first byte). - `TestSentenceBoundary` pins the boundary rule case by case. - glchat tests cover `/thinking` setting, keeping and clearing values, bare `/thinking` changing nothing, bad lines refused whole, and the budgets reaching the request body only when set. - Server tests cover passing both fields through (0 included) and rejecting negative values and soft > hard. - `go vet ./...` and `go test ./...` pass. - Served NVIDIA-Nemotron-3-Nano-30B-A3B-NVFP4 on the RTX PRO 6000, greedy, 6 reasoning prompts (word problems, counting, ordering, code) × hard budgets of 40, 70, 100 and 150; 22 of the 24 runs were cut (in the other two the model finished thinking within the budget): | closing sequence | answer continues the reasoning | answer starts cleanly | `reasoning_tokens` over budget | | --- | ---: | ---: | ---: | | bare `</think>` | 11 | 11 | 0 | | `\n</think>\n` (this PR) | 0 | 22 | 0 | Tight budgets still cost accuracy: some cut turns answered wrongly under either sequence, which no closing sequence can fix. - Also on the GPU: soft 64 on a 612-token reasoning turn cut after "= 6:10 pm.\n\n" at 84 tokens; a streamed request split cleanly into reasoning and content deltas; `response_format: json_object` with a hard budget of 20 returned 20 reasoning tokens and valid JSON; negative budgets and soft > hard returned 400. 🤖 Generated with [Claude Code](https://claude.com/claude-code)
feat(engine): soft and hard thinking-token budgets for reasoning turns
All checks were successful
ci / test_and_build (pull_request) Successful in 30s
0af68208e4
Chat requests take max_thinking_tokens_soft and max_thinking_tokens_hard.
Past the hard budget the engine emits the reasoning-end token in place of
the sampled one, closing the block at once; past the soft budget it does
so at the next sentence boundary (a newline, or whitespace after terminal
punctuation). The forced marker closes the block through the ordinary
reasoning split, and the model then writes its answer.

Budgets must be >= 0 with soft <= hard; a turn that does not reason
ignores them. The completion log line records the budgets and which one
cut the block.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Collaborator

Automated review by pr-reviewer v0.54.0 | Safety Check | Nemotron 3 Nano | tracking id r-c0ab64-8e1fe1
This is an AI-generated review and may contain mistakes.

Status: ❌ Failed


Review failed. Tracking id r-c0ab64-8e1fe1 — see logs for details.

Comment @pr-reviewer-bot retry to try again.

<!-- pr-reviewer:review --> *Automated review by [pr-reviewer](https://git.brooktrails.org/brooktrails/pr-reviewer) v0.54.0 | Safety Check | Nemotron 3 Nano | tracking id `r-c0ab64-8e1fe1`* *This is an AI-generated review and may contain mistakes.* **Status:** ❌ Failed --- Review failed. Tracking id `r-c0ab64-8e1fe1` — see logs for details. Comment `@pr-reviewer-bot retry` to try again.
fix(engine): treat a line break folded into punctuation as a sentence boundary
All checks were successful
ci / test_and_build (pull_request) Successful in 34s
de61e2c7f0
Byte-level vocabularies such as Nemotron 3's tokenize ".\n\n", "?\n" and
".\n" as single tokens, so the token after a sentence that ends a line
starts with neither whitespace nor a newline, and the soft thinking budget
skipped every such boundary. Text that already ends in a newline is now a
boundary too. On Nemotron 3 Nano a soft budget of 64 cut at token 103
before and at the next line end, token 84, after.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
fix(engine): close a budget-cut reasoning block the way the template does
All checks were successful
ci / test_and_build (pull_request) Successful in 34s
8d7e313042
A thinking budget closed the block with a bare reasoning-end token. Cut
mid-thought, Nemotron 3 Nano read that as noise and carried on reasoning
in its answer: across 6 prompts at hard budgets of 40-150 tokens it did
so in 11 of 22 cuts, once by emitting a call to a Python tool the request
never offered, and the leaked reasoning made those turns longer.

The engine now forces the chat template's own closing sequence,
"\n</think>\n", one token a step via Sequence.Forced. The leading newline
is skipped when the text already ends a line, and the trailing one under
a grammar, which constrains every token past the marker; a tokenizer that
cannot round-trip "\n" still gets the bare marker. Rerun on the same 22
cuts, the answer started cleanly every time.

The hard budget now reserves room for the leading newline, so
reasoning_tokens never exceeds it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
feat(glchat): /thinking sets reasoning and its token budgets
All checks were successful
ci / test_and_build (pull_request) Successful in 32s
ac4e5ae4b4
glchat could only choose whether a model reasons, and only at startup
(--think). /thinking changes it mid-conversation and sets the server's
reasoning budgets, for every turn from the next one on:

    /thinking on soft=512 hard=1024
    /thinking default hard=off

on, off and default set enable_thinking (default leaves it to the
template, as without --think); soft=N and hard=N set
max_thinking_tokens_soft and max_thinking_tokens_hard, and =off clears
one. Words left out keep their value, a line with any bad word is refused
whole, soft above hard is refused before it can fail a turn, and bare
/thinking reports the setting. Budgets stay off the wire until set.

The help and completion lists pad usage to a fixed column; a usage
longer than it now runs over with a gap instead of running into its
summary.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
rcsheets changed title from feat(engine): soft and hard thinking-token budgets for reasoning turns to feat: soft and hard thinking-token budgets for reasoning turns 2026-10-03 05:50:05 +00:00
feat(glchat): deprecate --think in favor of /thinking
Some checks failed
ci / test_and_build (pull_request) Has been cancelled
5e6087d4bc
/thinking sets the same enable_thinking choice mid-conversation, next to
the reasoning budgets, so the launch flag is now a second way to do part
of the same thing. --think keeps working; cobra hides it from --help and
prints a deprecation notice naming /thinking whenever it is used. It can
be removed in a later release.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
rcsheets force-pushed feat/thinking-budgets from 5e6087d4bc
Some checks failed
ci / test_and_build (pull_request) Has been cancelled
to 727afbe575
All checks were successful
ci / test_and_build (pull_request) Successful in 34s
2026-10-03 05:59:46 +00:00
Compare
docs: guidance for setting thinking budgets, from measured runs
All checks were successful
ci / test_and_build (pull_request) Successful in 30s
d6c7ca420e
Adds a Thinking budgets section to the README: what a leak looks like
(reasoning continuing in the answer, often ending in a stray </think>),
and three settings rules measured on Nemotron 3 Nano across 515
budget-cut turns sampled at temperature 1.0 -- pair every hard budget
with a soft one (hard alone at 64 leaked in 5 of 48 turns, any soft
budget below it in 3 of 390), leave 32 to 64 tokens between them (the
model reached a sentence boundary a median 11 tokens past the soft
budget), and expect accuracy to fall at tight budgets (48/48
unbudgeted, 29-38 at 64-128, 37-41 at 256).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
rcsheets deleted branch feat/thinking-budgets 2026-10-03 18:12:12 +00:00
Sign in to join this conversation.
No reviewers
No labels
No milestone
No project
No assignees
2 participants
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
brooktrails/gllm!110
No description provided.