docs: knobs for the operator's goals, and measuring what they do #111
Loading…
Reference in a new issue
No description provided.
Delete branch "docs/characterize-settings"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
gllm's conventions covered how automatic choices should work ("smart defaults, power-user overrides") but not why the knobs exist or how settings guidance should be arrived at. This adds that as an architecture convention in AGENTS.md.
The principle
A goal of gllm is to make it easy for whoever writes the
gllm servecommand to get what they want out of the inference it serves: fewer tokens, lower latency, higher accuracy, less energy, no failure modes the client must clean up. So gllm should expose the knobs that change how the model does inference, and be able to characterize each one: its effect on accuracy, length, latency, throughput and energy, and which failures it causes.What it asks of new knobs
reasoning_cutdoes), so outcomes can be attributed.It also records the harness lessons from that work: save results incrementally, pair runs across settings, sample at the model card's settings, and expect each model family to fail in its own way.
The README status checklist gains an open item for the characterization harness.
🤖 Generated with Claude Code
Automated review by pr-reviewer v0.54.0 | Safety Check | Nemotron 3 Nano | tracking id
r-c15268-fa4893This is an AI-generated review and may contain mistakes.
Status: ❌ Failed
Review failed. Tracking id
r-c15268-fa4893— see logs for details.Comment
@pr-reviewer-bot retryto try again.