brooktrails/gllm
Watch
1
0
Fork
You've already forked gllm
 0
Code Issues 1 Pull requests Projects Releases Packages Wiki Activity Actions

test(grammar): give the walk vocabulary number-shaped tokens #61

Merged
rcsheets merged 1 commit from feat/grammar-schema-keywords into main 2026-08-04 22:53:17 +00:00
Conversation 1 Commits 1 Files changed 1 +8
rcsheets commented 2026-08-04 22:50:59 +00:00
Owner
Copy link

walkVocab exists to carry the acceptor across a schema decision point in
a single step -- a token is allowed only if every one of its bytes is, so
one can commit to a position the acceptor never got to reconsider. Its
tokens covered keys, separators and closers, but numbers only as the
single digit 0, which predates numeric bounds being enforced at all.

Numbers are now a decision point of their own: 12 can carry a prefix
from feasible to unreachable without the acceptor ever seeing the 1
alone, and e- or .5 commits to a shape that freezes what the
remaining digits can reach. Those are the steps where the no-dead-end
property is hardest to hold, and none of them were reachable before.

No failures; the walk finds new states, which is the point.

Co-Authored-By: Claude Opus 5 noreply@anthropic.com

walkVocab exists to carry the acceptor across a schema decision point in a single step -- a token is allowed only if every one of its bytes is, so one can commit to a position the acceptor never got to reconsider. Its tokens covered keys, separators and closers, but numbers only as the single digit `0`, which predates numeric bounds being enforced at all. Numbers are now a decision point of their own: `12` can carry a prefix from feasible to unreachable without the acceptor ever seeing the `1` alone, and `e-` or `.5` commits to a shape that freezes what the remaining digits can reach. Those are the steps where the no-dead-end property is hardest to hold, and none of them were reachable before. No failures; the walk finds new states, which is the point. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
test(grammar): give the walk vocabulary number-shaped tokens
All checks were successful
ci / test_and_build (pull_request) Successful in 21s
Details
4d569f106e 
walkVocab exists to carry the acceptor across a schema decision point in
a single step -- a token is allowed only if every one of its bytes is, so
one can commit to a position the acceptor never got to reconsider. Its
tokens covered keys, separators and closers, but numbers only as the
single digit `0`, which predates numeric bounds being enforced at all.

Numbers are now a decision point of their own: `12` can carry a prefix
from feasible to unreachable without the acceptor ever seeing the `1`
alone, and `e-` or `.5` commits to a shape that freezes what the
remaining digits can reach. Those are the steps where the no-dead-end
property is hardest to hold, and none of them were reachable before.

No failures; the walk finds new states, which is the point.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
pr-reviewer-bot commented 2026-08-04 22:51:01 +00:00
Collaborator
Copy link

Automated review by pr-reviewer v0.44.2 | Safety Check | Claude Sonnet 5 | tracking id r-726cd5-76611a
This is an AI-generated review and may contain mistakes.

Status: ✅ Completed

⚠️ The primary review (Mistral Small) failed. The review below is from Claude Sonnet 5, which reviewed the same diff.

✅ Verdict: LGTM — The change only adds test vocabulary tokens with no production code impact, so nothing alarming stands out.

This is a small, self-contained test-only change that adds new multi-byte number-shaped tokens to a fake tokenizer's vocabulary used for exhaustive walk testing. It's additive (no removed tokens), doesn't touch production logic, and the comment explains the rationale clearly. No secrets, no binaries, no breaking changes — just new test fixtures to exercise previously-unreached acceptor states.

*Automated review by [pr-reviewer](https://git.brooktrails.org/brooktrails/pr-reviewer) v0.44.2 | Safety Check | Claude Sonnet 5 | tracking id `r-726cd5-76611a`* *This is an AI-generated review and may contain mistakes.* **Status:** ✅ Completed --- > ⚠️ The primary review (**Mistral Small**) failed. The review below is from **Claude Sonnet 5**, which reviewed the same diff. **✅ Verdict: LGTM** — The change only adds test vocabulary tokens with no production code impact, so nothing alarming stands out. This is a small, self-contained test-only change that adds new multi-byte number-shaped tokens to a fake tokenizer's vocabulary used for exhaustive walk testing. It's additive (no removed tokens), doesn't touch production logic, and the comment explains the rationale clearly. No secrets, no binaries, no breaking changes — just new test fixtures to exercise previously-unreached acceptor states.
rcsheets deleted branch feat/grammar-schema-keywords 2026-08-04 22:53:17 +00:00
Sign in to join this conversation.
Reviewers
No reviewers
Labels
Clear labels
No items
No labels
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
2 participants
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
brooktrails/gllm!61
No description provided.