feat(engine): response debug preview, unlimited previews in files, and closing lines for unfinished requests #107

Merged
rcsheets merged 3 commits from feat/debug-response-preview into main 2026-10-02 08:17:05 +00:00
Owner

gllm serve already has --debug-with-prompt-preview, which adds the text of the first few prompt tokens to the debug engine: request received line. This adds the response-side counterpart, lets either preview be unlimited, and gives requests that do not complete a closing log line.

Response preview

--debug-with-response-preview adds a response_preview field to the engine: request complete line: the text of the first few generated tokens, rendered the same way as the prompt preview (control tokens by name rather than dropped). --debug-response-preview-size=N sets how many tokens (default 8). It is off by default, since it logs response content.

The request-complete line is INFO, but the preview is a debug aid, so like the prompt preview it only appears at --log-level debug.

Unlimited previews

--debug-prompt-preview-size=-1 and --debug-response-preview-size=-1 mean the whole prompt or response. That is too much for a log line, so an unlimited preview is written to a file named for the request id, and the log line carries the path in place of the preview field:

Preview File Log field
prompt <request_id>.prompt.txt prompt_file on engine: request received
response <request_id>.response.txt response_file on the request's closing line

--debug-preview-dir names the directory. Without it, the engine makes a fresh directory with os.MkdirTemp -- gllm-previews-* in $TMPDIR, else /tmp -- on first use, and logs its path once. A fixed name in a shared temp dir would be predictable, so another user could create it first and read or redirect what is written; MkdirTemp picks an unused name and makes the directory owner-only. Files are written 0600. Request ids are minted by the engine, never taken from the client, so they are safe file names.

Nothing prunes the directory. A failed write is logged as a warning and does not fail the request.

Closing lines for cancelled and failed requests

Every request that reaches the engine has an id -- the server mints it and sends it in X-Request-Id before generating -- but only a completed request logged it at the end. A cancelled request logged nothing after request received; a failed one logged at most the server's ERROR line, which has no id, and a too-large or not-implemented request logged nothing at all.

Each now gets an INFO closing line with request_id, prompt_tokens, completion_tokens and err:

  • engine: request cancelled when the request context ended (the client went away).
  • engine: request failed for any other error. A prompt that fails to tokenize gets this line too, with only request_id and err, since it was never admitted.

These lines are INFO whatever the error: the server still decides what deserves its ERROR line. Both carry the response preview the same way request complete does, so a stopped request shows what it had generated -- and with an unlimited preview, every admitted request ends up with a response file, empty if it produced nothing. cancel returns the generated tokens, copied under the engine lock, for this.

Requests rejected before an id is minted (an undecodable body, a wrong model name) are unchanged: no id, no closing line, no files.

Behaviour changes

  • A preview size of 0 still means "use the default", but a negative size, which used to be treated the same way, now means unlimited.
  • Cancelled and failed requests add one INFO line each to the log at the default log level.

Testing

go build ./..., go vet ./..., gofmt and go test ./... are clean, and the engine and server tests pass under -race. New engine tests cover the response preview and its size cap, its gating on the option and the debug level, unlimited previews written to a named directory, the default temp directory (one per engine, mode 0700, logged once), and the cancelled and failed closing lines with their response files. Not exercised against a real checkpoint or on a GPU.

🤖 Generated with Claude Code

`gllm serve` already has `--debug-with-prompt-preview`, which adds the text of the first few prompt tokens to the debug `engine: request received` line. This adds the response-side counterpart, lets either preview be unlimited, and gives requests that do not complete a closing log line. ## Response preview `--debug-with-response-preview` adds a `response_preview` field to the `engine: request complete` line: the text of the first few generated tokens, rendered the same way as the prompt preview (control tokens by name rather than dropped). `--debug-response-preview-size=N` sets how many tokens (default 8). It is off by default, since it logs response content. The request-complete line is INFO, but the preview is a debug aid, so like the prompt preview it only appears at `--log-level debug`. ## Unlimited previews `--debug-prompt-preview-size=-1` and `--debug-response-preview-size=-1` mean the whole prompt or response. That is too much for a log line, so an unlimited preview is written to a file named for the request id, and the log line carries the path in place of the preview field: | Preview | File | Log field | | --- | --- | --- | | prompt | `<request_id>.prompt.txt` | `prompt_file` on `engine: request received` | | response | `<request_id>.response.txt` | `response_file` on the request's closing line | `--debug-preview-dir` names the directory. Without it, the engine makes a fresh directory with `os.MkdirTemp` -- `gllm-previews-*` in `$TMPDIR`, else `/tmp` -- on first use, and logs its path once. A fixed name in a shared temp dir would be predictable, so another user could create it first and read or redirect what is written; `MkdirTemp` picks an unused name and makes the directory owner-only. Files are written 0600. Request ids are minted by the engine, never taken from the client, so they are safe file names. Nothing prunes the directory. A failed write is logged as a warning and does not fail the request. ## Closing lines for cancelled and failed requests Every request that reaches the engine has an id -- the server mints it and sends it in `X-Request-Id` before generating -- but only a completed request logged it at the end. A cancelled request logged nothing after `request received`; a failed one logged at most the server's ERROR line, which has no id, and a too-large or not-implemented request logged nothing at all. Each now gets an INFO closing line with `request_id`, `prompt_tokens`, `completion_tokens` and `err`: - `engine: request cancelled` when the request context ended (the client went away). - `engine: request failed` for any other error. A prompt that fails to tokenize gets this line too, with only `request_id` and `err`, since it was never admitted. These lines are INFO whatever the error: the server still decides what deserves its ERROR line. Both carry the response preview the same way `request complete` does, so a stopped request shows what it had generated -- and with an unlimited preview, every admitted request ends up with a response file, empty if it produced nothing. `cancel` returns the generated tokens, copied under the engine lock, for this. Requests rejected before an id is minted (an undecodable body, a wrong model name) are unchanged: no id, no closing line, no files. ## Behaviour changes - A preview size of `0` still means "use the default", but a negative size, which used to be treated the same way, now means unlimited. - Cancelled and failed requests add one INFO line each to the log at the default log level. ## Testing `go build ./...`, `go vet ./...`, `gofmt` and `go test ./...` are clean, and the engine and server tests pass under `-race`. New engine tests cover the response preview and its size cap, its gating on the option and the debug level, unlimited previews written to a named directory, the default temp directory (one per engine, mode 0700, logged once), and the cancelled and failed closing lines with their response files. Not exercised against a real checkpoint or on a GPU. 🤖 Generated with [Claude Code](https://claude.com/claude-code)
The response-side counterpart of --debug-with-prompt-preview: at
--log-level debug it adds a response_preview field (the text of the first
few generated tokens, control tokens shown by name) to the
"engine: request complete" line. --debug-response-preview-size sets how
many tokens (default 8). Off by default, since it logs response content.

The token rendering both previews share moves into Engine.tokenPreview.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
feat(engine): allow unlimited debug previews, written to per-request files
All checks were successful
ci / test_and_build (pull_request) Successful in 34s
5b6457a6c0
A preview size of -1 now means the whole prompt or response rather than
its first few tokens. That is too much for a log line, so an unlimited
preview is written to <request id>.prompt.txt or <request id>.response.txt
and the log line carries the path (prompt_file, response_file) in place of
the preview field.

--debug-preview-dir names the directory. Without it the engine makes a
fresh owner-only gllm-previews-* directory with os.MkdirTemp in $TMPDIR
(else /tmp) on first use and logs its path: a fixed name in a shared temp
dir would be predictable, letting another user create it first.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Collaborator

Automated review by pr-reviewer v0.52.3 | Safety Check | Nemotron 3 Nano | tracking id r-bf65c4-7ef5fb
This is an AI-generated review and may contain mistakes.

Status: ❌ Failed


Review failed. Tracking id r-bf65c4-7ef5fb — see logs for details.

Comment @pr-reviewer-bot retry to try again.

<!-- pr-reviewer:review --> *Automated review by [pr-reviewer](https://git.brooktrails.org/brooktrails/pr-reviewer) v0.52.3 | Safety Check | Nemotron 3 Nano | tracking id `r-bf65c4-7ef5fb`* *This is an AI-generated review and may contain mistakes.* **Status:** ❌ Failed --- Review failed. Tracking id `r-bf65c4-7ef5fb` — see logs for details. Comment `@pr-reviewer-bot retry` to try again.
feat(engine): log a closing line for cancelled and failed requests
All checks were successful
ci / test_and_build (pull_request) Successful in 34s
865250bd9e
A request that did not complete left its id unaccounted for: a cancelled
one logged nothing after "request received", and a failed one at most an
ERROR line without the id. Each now gets an INFO closing line carrying
request_id, the token counts and err -- "engine: request cancelled" when
the client went away, "engine: request failed" otherwise, including a
prompt that would not tokenize.

Both carry the response preview like "request complete" does, so a
stopped request shows (or, when unlimited, writes to its response file)
what it had generated. cancel returns that output, copied under the engine
lock.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
rcsheets changed title from feat(engine): add a response debug preview, and unlimited previews written to files to feat(engine): response debug preview, unlimited previews in files, and closing lines for unfinished requests 2026-10-02 08:05:37 +00:00
rcsheets deleted branch feat/debug-response-preview 2026-10-02 08:17:05 +00:00
Sign in to join this conversation.
No reviewers
No labels
No milestone
No project
No assignees
2 participants
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
brooktrails/gllm!107
No description provided.