docs: knobs for the operator's goals, and measuring what they do #111

Merged
rcsheets merged 1 commit from docs/characterize-settings into main 2026-10-03 20:31:09 +00:00
Owner

gllm's conventions covered how automatic choices should work ("smart defaults, power-user overrides") but not why the knobs exist or how settings guidance should be arrived at. This adds that as an architecture convention in AGENTS.md.

The principle

A goal of gllm is to make it easy for whoever writes the gllm serve command to get what they want out of the inference it serves: fewer tokens, lower latency, higher accuracy, less energy, no failure modes the client must clean up. So gllm should expose the knobs that change how the model does inference, and be able to characterize each one: its effect on accuracy, length, latency, throughput and energy, and which failures it causes.

  • Guidance is measured, not asserted, and the docs state the method (model, task set, sampling, sample size, how the metric was checked). The README's Thinking budgets section, from #110, is the reference example.
  • The direction is automation: gllm, or a harness shipped beside it (glbench being the natural home), runs these sweeps and suggests settings for a goal the operator states.

What it asks of new knobs

  • Per-request where safe, with the server flag as the default, so one server can be swept without restarts.
  • Report what the knob did on each request (as the completion log's reasoning_cut does), so outcomes can be attributed.
  • Outcome metrics a program can check, validated against hand labels before being trusted. In the thinking-budget work, a model judging its own output was unreliable: Nemotron missed 8 of 11 leaks.

It also records the harness lessons from that work: save results incrementally, pair runs across settings, sample at the model card's settings, and expect each model family to fail in its own way.

The README status checklist gains an open item for the characterization harness.

🤖 Generated with Claude Code

gllm's conventions covered *how* automatic choices should work ("smart defaults, power-user overrides") but not *why* the knobs exist or how settings guidance should be arrived at. This adds that as an architecture convention in AGENTS.md. ## The principle A goal of gllm is to make it easy for whoever writes the `gllm serve` command to get what they want out of the inference it serves: fewer tokens, lower latency, higher accuracy, less energy, no failure modes the client must clean up. So gllm should expose the knobs that change how the model does inference, and be able to characterize each one: its effect on accuracy, length, latency, throughput and energy, and which failures it causes. - **Guidance is measured, not asserted,** and the docs state the method (model, task set, sampling, sample size, how the metric was checked). The README's *Thinking budgets* section, from #110, is the reference example. - **The direction is automation:** gllm, or a harness shipped beside it (glbench being the natural home), runs these sweeps and suggests settings for a goal the operator states. ## What it asks of new knobs - Per-request where safe, with the server flag as the default, so one server can be swept without restarts. - Report what the knob did on each request (as the completion log's `reasoning_cut` does), so outcomes can be attributed. - Outcome metrics a program can check, validated against hand labels before being trusted. In the thinking-budget work, a model judging its own output was unreliable: Nemotron missed 8 of 11 leaks. It also records the harness lessons from that work: save results incrementally, pair runs across settings, sample at the model card's settings, and expect each model family to fail in its own way. The README status checklist gains an open item for the characterization harness. 🤖 Generated with [Claude Code](https://claude.com/claude-code)
docs: knobs for the operator's goals, and measuring what they do
All checks were successful
ci / test_and_build (pull_request) Successful in 30s
6d2ff6bd19
Records a development principle in AGENTS.md: gllm should make it easy
for whoever writes the `gllm serve` command to get what they want from
the inference, by exposing knobs that change how the model does
inference and by characterizing what each one does. Settings guidance is
measured rather than asserted, with the method stated, and the long-term
direction is for gllm or a harness beside it (glbench) to run those
sweeps and suggest settings for a stated goal. The convention lists what
that asks of new knobs: per-request where safe, per-request reporting of
their effect, outcome metrics a program can check and that have been
validated against hand labels.

The README status checklist gains the open item for that harness.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Collaborator

Automated review by pr-reviewer v0.54.0 | Safety Check | Nemotron 3 Nano | tracking id r-c15268-fa4893
This is an AI-generated review and may contain mistakes.

Status: ❌ Failed


Review failed. Tracking id r-c15268-fa4893 — see logs for details.

Comment @pr-reviewer-bot retry to try again.

<!-- pr-reviewer:review --> *Automated review by [pr-reviewer](https://git.brooktrails.org/brooktrails/pr-reviewer) v0.54.0 | Safety Check | Nemotron 3 Nano | tracking id `r-c15268-fa4893`* *This is an AI-generated review and may contain mistakes.* **Status:** ❌ Failed --- Review failed. Tracking id `r-c15268-fa4893` — see logs for details. Comment `@pr-reviewer-bot retry` to try again.
rcsheets deleted branch docs/characterize-settings 2026-10-03 20:31:09 +00:00
Sign in to join this conversation.
No reviewers
No labels
No milestone
No project
No assignees
2 participants
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
brooktrails/gllm!111
No description provided.