copydesk

Sits in between raw LLM output and the consumer, in a genuine attempt to improve response quality.

copydesk is an OpenAI-compatible proxy that fact-checks the model behind it. It fronts one or more OpenAI-compatible inference endpoints, presents the pipeline as an ordinary model ID (mistral-7b-edited), and only releases a response after its factual claims have been extracted, checked against retrieved evidence, and corrected or hedged as needed. The copydesk is the newspaper desk where copy sits between reporter and press; nothing prints without passing it.

Slower and more expensive than the model it wraps, by design. Clients that want the fast path keep talking to the upstream model directly.

How a request moves through the desk

client ── /v1/chat/completions ──> copydesk
  1. draft        upstream call to the writer model (gllm / Mistral 7B);
                  requests always pass through unchanged -- copydesk
                  inspects responses, never prompts
  2. mechanical   truncation (finish_reason), notation/format sanity
  3. extract      cheap first-token check: does this response contain
                  factual claims at all? (no -> release as "wire copy");
                  then atomic claims as {claim, verbatim_span,
                  open_question, query}; spans validated mechanically
                  against the draft
  4. triage       check-worthiness filter; skip claims not worth a
                  retrieval round-trip (zero survivors -> wire copy)
  5. retrieve     local corpus first (pgvector), web fallback optional
  6. verify       claim vs. passages: supports / contradicts / not
                  addressed (three-way; silence is not confirmation).
                  Empty retrieval falls back to k-sample self-consistency
  7. route        span-local errors -> surgical string substitution;
                  structural errors -> constrained regeneration with the
                  verified facts injected; split votes -> hedge or delete,
                  never assert the majority
  8. respond      final message, with the verification trace emitted as
                  reasoning content so the wait is legible and auditable

Gating on the response rather than the prompt is deliberate: a prompt-side gate predicts, and its false negatives ship unverified claims under the edited model ID. A response-side gate observes; its worst case is wasted latency. Anyone talking to mistral-7b-edited has already chosen slower-but-checked, so there is no fast lane to protect.

The verifier model may differ from the writer (asymmetric checking: 7B drafts, Mistral Small verifies). The verifier sees only the claim and the passages -- never the original response or conversation -- to avoid anchoring toward agreement.

Status

Roughly in dependency order:

• OpenAI-compatible surface: /v1/models, /v1/chat/completions (non-streaming first), upstream client with per-stage model routing
• wire-copy short-circuit (response-side first-token "any claims?" check at the front of extraction)
• mechanical pass: finish_reason/truncation detection, format sanity
• claim extraction with span validation (drop extractions whose verbatim_span does not appear verbatim in the draft)
• check-worthiness triage (port prompts from Loki/FacTool rather than authoring from scratch)
• corpus ingestion job: dump -> chunk -> embed -> pgvector (CNPG)
• retrieval + three-way verification against passages, supporting sentence emitted verbatim for auditability
• self-consistency fallback (k-sample majority vote on the open_question form of the claim; split votes mean "doesn't know")
• correction routing: surgical substitution vs. constrained regeneration, heuristic classifier between them
• structured-claim tier: enumerable facts (lookup-table shaped) checked by query, not by model
• verification trace as reasoning content (SSE), keepalives, then real streaming of the final message
• stet ledger: per-request record of every claim, verdict, evidence snippet, and action taken (Postgres)
• Prometheus metrics: per-stage latency, claims extracted/checked/ corrected/hedged, upstream token spend
• convergence guard (single pass by default; loop capped at 2)
• asymmetric verifier config (writer and verifier as distinct upstreams)

Layout

cmd/copydesk/           server binary
internal/server/        OpenAI-compatible HTTP surface
internal/pipeline/      stage orchestration; ties everything together
internal/extract/       wire-copy short-circuit, claim extraction +
                        span validation
internal/triage/        check-worthiness
internal/retrieve/      pgvector corpus, optional web fallback
internal/verify/        three-way verification, self-consistency voting
internal/route/         surgical substitution vs. constrained regen
internal/upstream/      OpenAI-compatible client (writer + verifier)
internal/corpus/        ingestion: dump -> chunk -> embed
internal/ledger/        stet ledger (Postgres)

Processor / retriever / verifier are interfaces (after OpenFactCheck's decomposition) so implementations stay swappable.

Design decisions

• Recoverability over cleverness. copydesk failing must not take the plain model down: it is a separate service with a separate blast radius. If Postgres or retrieval is unavailable it degrades to pass-through, prepending a visible warning to the response ("Editors were not able to review the following for factual accuracy:") and noting the same in the trace, so an unedited answer never masquerades as an edited one.
• Silence is not confirmation. "Not addressed" is a real verdict. Claims that retrieval cannot reach and self-consistency cannot settle get hedged or deleted, never asserted.
• The verifier must be allowed to lose. Every override is recorded in the stet ledger with its evidence, so bad corrections are auditable and the desk's own error rate is measurable.
• Postgres for everything stateful (corpus via pgvector, ledger as plain tables). No second datastore.
• Prompts are ported, not authored. Claim decomposition and verification prompts start from published, tested ones (Loki, FacTool, Factcheck-GPT) and diverge only with eval evidence.

Non-goals

• Not an inference engine. Tokens-in-tokens-out belongs to gllm/vLLM; copydesk answers a different question ("what response are we willing to stand behind"), and no kernel, sampler, or KV-cache code lives here. (Grammar-constrained JSON decoding, which extraction wants badly, is a gllm feature, not a copydesk one.)
• Not a guardrails framework. No policy engine, no jailbreak detection, no configuration DSL. One pipeline, done well.
• Not an after-the-fact checker for arbitrary text (Loki et al. already exist). copydesk lives in the serving path or not at all.

Prior art

optillm proves the optimizing-proxy topology; Loki / FacTool / Factcheck-GPT / OpenFactCheck prove the claim-extract-retrieve-verify pipeline. Neither cluster combines the proxy facade with retrieval-grounded verification and correction routing, and neither is Go. That intersection is this project.

Running

copydesk serve \
  --addr :8100 \
  --writer   http://gllm.c2-talos:8000/v1  --writer-model mistral-7b \
  --verifier http://vllm.c2-talos:8000/v1  --verifier-model mistral-small \
  --postgres postgres://copydesk@cnpg-rw:5432/copydesk \
  --serve-as mistral-7b-edited

curl localhost:8100/v1/models