SLP - Simple LLM Proxy

An LLM Proxy that is not very smart. Presents one OpenAI-compatible surface in front of several OpenAI-compatible backends, routes each request to exactly one of them by model ID, and is otherwise too dumb to touch the bytes in flight.

SLP is copydesk's counterpart on the same shelf: copydesk is the editor who reads every line before it prints; SLP is the runner who carries the copy to the right desk and is constitutionally incapable of reading it. Every feature it does not have is a class of bug that cannot corrupt a response.

It exists because routing on the model field means looking inside the request body, which Caddy and nginx will not do without a custom module -- and once a module is parsing the body it has stopped being config and started being a program. SLP is that program, kept as small as the job allows. Put it behind Caddy: TLS, HTTP/2, and ACME stay Caddy's job; body-aware routing is SLP's.

What a request does

client ── POST /v1/chat/completions {"model":"mistral-7b-edited"} ──> SLP
  1. read     buffer the (small) request body
  2. look up  model id -> upstream, from the compiled route table
  3. rename   if the route declares one, swap in the upstream's own model
              name -- the only field SLP ever writes (see "Model names")
  4. forward  reverse-proxy to that upstream, byte for byte
              (FlushInterval = -1: SSE frames and reasoning_content pass
               through untouched; SLP never parses the response)

No pools, no failover. GET /v1/models is answered from the route table directly -- SLP never fans out on the request path, so a slow upstream (copydesk mid-verification) cannot stall the catalog. The one place SLP talks to a backend unprompted is model-name discovery, which runs on its own schedule in the background and never blocks a request or the catalog.

Routing is by exact model id

One model id maps to one upstream. SLP does not alias, group, or load-balance ids. This is deliberate: mistral-7b and mistral-7b-edited are different products with different latency contracts -- the whole point of the edited path is that it is not interchangeable with the raw one -- so they stay distinct routes to distinct upstreams. A backend that wants two dials behind one id is a smarter program's job.

Model names: the exposed id is not always the upstream's id

The id SLP exposes is a product name you choose. The id a backend answers to is whatever it was loaded with, which is frequently a build artifact:

{"error":{"message":"model \"mistral-small-4\" does not exist;
                     this server serves \"Mistral-Small-4-119B-2603-NVFP4\""}}

A client that reads GET /v1/models and echoes back what it saw will hit this every time, so the two names have to be reconciled somewhere. Renaming the route to match the backend leaks a quantization format into the catalog and breaks the moment the weights are rebuilt. SLP reconciles it instead, by asking the backend what it calls itself.

Each raw backend gets a supervisor goroutine that probes its /v1/models. If the backend serves exactly one model -- the normal case for a single-model vLLM or gllm -- that name becomes the route's rewrite rule, and the model field is swapped on the way past. Nothing else in the body is touched.

client sends  {"model":"mistral-small-4", ...}
upstream gets {"model":"Mistral-Small-4-119B-2603-NVFP4", ...}

Discovery never gates startup. The supervisors run alongside the server, not before it. A backend that is down, or a vLLM still loading a 119B model, is retried indefinitely (1s backing off to 30s) without SLP waiting on it; the route simply forwards unchanged until a name is known, which is exactly what it would do if discovery did not exist. A backend outage cannot become an SLP outage, and boot order in the cluster stops mattering.

Once a name is known the supervisor backs off to a five-minute check and is only watching for the name to change. When a weights rebuild renames the model under a running SLP, the rewrite rule follows it -- no restart, no config edit.

Ambiguity never guesses. If a backend serves zero or several models, SLP cannot know which one the route means, so it keeps the last known good name (or forwards unchanged if it never had one) and says so loudly:

discovery: WARNING: mistral-small-4: backend serves 2 models [model-a model-b],
  expected exactly 1; keeping last known name "Mistral-Small-4-119B-2603-NVFP4"
  -- requests may be routed to a stale model

with slp_upstream_model_ambiguous{model_id="..."} going to 1 for the duration. That gauge is the one to alert on: it is the only state where SLP keeps serving using a name it can no longer confirm. Also exported: slp_discovery_probes_total{model_id,result} (ok/ambiguous/error), slp_upstream_model_changes_total, and slp_upstream_model_info, which carries the live mapping in its labels.

Pinning a name

An explicit model: on a backend overrides discovery and skips probing entirely -- for a multi-model backend that discovery structurally cannot disambiguate, an upstream that does not serve /v1/models, or a name you do not want drifting:

backends:
  - name: mistral-small-4                      # what clients see and send
    url: http://vllm.c2-talos:8000             # discovered; nothing to declare
    timeout: 60s
  - name: llama-70b
    url: http://gllm-2.c2-talos:8000
    model: Llama-3.3-70B-Instruct-AWQ          # pinned; never probed
    timeout: 120s

Startup logs show which routes are pinned and which are still resolving:

llama-70b       -> http://gllm-2.c2-talos:8000 as "Llama-3.3-70B-Instruct-AWQ"
mistral-small-4 -> http://vllm.c2-talos:8000 (discovering)

Editions are never probed. A copydesk instance is launched as --serve-as mistral-small-4-edited and expects the exposed id, so editions forward it unchanged unless they declare their own model: template (over .Name, .URL, .Model) -- useful for a pass-through wrapper that hands the body to the real backend.

Asking what an id resolves to: GET /v1/models?debug

Startup logs and slp_upstream_model_info both carry the live mapping, but neither is at hand when a client is being told its model does not exist. ?debug answers from the same route table the plain catalog is built from, still without touching a backend:

{"object": "list", "data": [
  {"id": "mistral-small-4",        "upstream_model": "Mistral-Small-4-119B-2603-NVFP4",
   "discovery": "discovered", "last_probe": "ok",
   "last_probe_at": "2026-07-27T09:14:02Z",
   "timeout": "1m0s", "retry": false, "aggregate": false},
  {"id": "llama-70b",              "discovery": "pending",
   "timeout": "2m0s", "retry": true,  "aggregate": false},
  {"id": "qwen-32b",               "upstream_model": "Qwen3-32B-AWQ",
   "discovery": "pinned",
   "timeout": "1m30s", "retry": false, "aggregate": false},
  {"id": "mistral-small-4-edited", "discovery": "passthrough",
   "timeout": "5m0s", "retry": false, "aggregate": true}
]}

discovery says where the name came from, which is the fast way to read a missing upstream_model: discovered (probed and current), pending (still probing, forwarding the exposed id meanwhile), pinned (declared in config, never probed), passthrough (an edition, serving the exposed id). ?debug, ?debug=1, and ?debug=yes all work; only an explicit false or 0 is a no.

The upstream URL is not in there. /v1/models is unauthenticated -- it is the one endpoint a client is supposed to read -- and naming backend hosts would undo the reticence the error path already keeps, where a dial failure tells the log the host, port, and resolved IP and tells the client "upstream request failed". Probe failures are the same: last_probe says error, and the log says which address refused. The debug catalog answers "what does SLP send as the model name", not "where does SLP send it".

Without the parameter the response is unchanged -- ids and nothing else, so an OpenAI client parses it exactly as before.

Editions: declare the transform once

The one place SLP is allowed to be clever is config, and only at load time. If you have three backends and want a copyedited variant of each, you should not have to write the same upstream three times.

An edition is a derived model id (base name + suffix) pointing at a derived upstream (a URL template over the backend's fields). It compiles, before the server starts, into ordinary flat routes. The runtime never sees an edition -- only the expanded table.

listen: ":8080"
log_requests: false      # true logs every forwarded request, not just failures

backends:
  - name: mistral-7b
    url: http://gllm.c2-talos:8000        # origin only; SLP forwards the request path verbatim
    timeout: 60s
  - name: mistral-small
    url: http://vllm.c2-talos:8000
    timeout: 60s
  - name: llama-70b
    url: http://gllm-2.c2-talos:8000
    timeout: 120s

editions:
  - suffix: -edited
    url: http://copydesk-{{.Name}}.c2-talos:8100   # template over backend fields: .Name, .URL
    timeout: 300s
    retry: false
    apply_to: ["*"]        # or a subset: [mistral-7b, llama-70b]

That compiles to six routes:

mistral-7b            -> http://gllm.c2-talos:8000
mistral-small         -> http://vllm.c2-talos:8000
llama-70b             -> http://gllm-2.c2-talos:8000
mistral-7b-edited     -> http://copydesk-mistral-7b.c2-talos:8100
mistral-small-edited  -> http://copydesk-mistral-small.c2-talos:8100
llama-70b-edited      -> http://copydesk-llama-70b.c2-talos:8100

The exposed id (mistral-7b-edited) is exactly what the corresponding copydesk instance is launched with (--serve-as mistral-7b-edited), so the two agree by construction. SLP knows nothing about copydesk specifically; an edition is generic, and copydesk is just its first consumer. A -json edition through some grammar-constrained wrapper would be the same shape.

Compilation fails closed. Duplicate model ids, an edition naming a backend that does not exist, or a template that does not resolve are startup errors, not runtime surprises. SLP either serves a fully-resolved table or refuses to start; there is no third state.

timeout is time-to-first-byte, not a ceiling

Per-route timeout is the response-header timeout -- how long SLP waits for the upstream to begin responding -- not a cap on the whole exchange. copydesk can spend five minutes on retrieval and verification, and a streaming backend can stream for as long as it likes; SLP only gives up if nothing has come back at all. This is why editions get a generous value.

When a route does time out, the log and the 502 body both name the length that expired, so you know which timeout: to raise without going back to config:

[slpq2ff15ba63230a0d1ddec5efe] Attempt 1/1 for model llama-70b failed: timed out after 120s: Post "http://gllm-2.c2-talos:8000/v1/chat/completions": net/http: timeout awaiting response headers

aggregate: streaming upstream on the client's behalf

timeout being time-to-first-byte has one blind spot: a non-streaming completion sends no bytes at all -- headers included -- until the last token is generated. On a slow model the whole generation happens inside the response-header timeout, and the knob stops working: sized for the worst-case generation it no longer detects a hung backend, sized as a liveness check it kills legitimate requests.

aggregate: true on a route closes the gap by making the upstream request a streaming one regardless of what the client asked for. When the client wanted plain JSON, SLP drains the SSE stream itself and reassembles the non-streaming response: deltas concatenated, usage, finish_reason, and gllm metadata carried over from the final chunk. Upstream headers now arrive at time-to-first-token, so timeout is a liveness check again and generation may take as long as it takes.

backends:
  - name: mistral-small-4
    url: http://newton.c2.brooktrails.us:8000
    timeout: 120s        # now bounds time-to-first-token, not the generation
    aggregate: true

This is SLP's one deliberate exception to never touching response bytes, which is why it is opt-in per route. The edges are handled conservatively:

• A client that asked to stream is untouched -- pass-through as always.
• Requests in the logprobs family pass through unconverted (gllm rejects logprobs with streaming); they keep the whole-generation timeout exposure, acceptable for a rare debugging shape.
• An upstream error status, or a backend that ignores stream: true, passes through byte for byte.
• A late in-band error event -- streaming commits its 200 before generation finishes, so failures arrive as an event -- is handed back as a 502 with the upstream's error object. A plain pass-through stream cannot do that; the aggregator can because it has written nothing to the client yet.
• A stream that dies before data: [DONE] is a 502, never a silently truncated 200.

usage in the reassembled response depends on the upstream putting usage on its final chunk, as gllm does. An upstream that never sends it in-stream yields responses without a usage field.

aggregate on an edition is its own declaration, never inherited from the backend: the edition's upstream is a different program with its own streaming behavior.

logging forwarded requests

Failures are always logged. Successful forwards are not, since that is a line per completion -- turn them on with log_requests: true at the top level of the config, or --log-requests for a one-off run:

[slpq2ff15ba63230a0d1ddec5efe] POST /v1/chat/completions: mistral-small-4 as "Mistral-Small-4-119B-2603-NVFP4" -> http://vllm.c2-talos:8000 200 in 1.482s (412 B up, 8317 B down) upstream=gdqz3f8a1c04b7e25d918a6f2c3b

The latency is time to first byte, the same measure as slp_upstream_latency_seconds -- for a streaming response it is when the stream started, not when it finished.

errors: the log gets the detail, the client gets the fact

A transport failure names the upstream's hostname, port, and resolved IP:

Post "http://gllm-2.c2-talos:8000/v1/chat/completions": dial tcp 10.69.60.58:8000: connect: connection refused

That is the route table talking, and the route table is SLP's business. A client behind Caddy generally cannot reach those addresses, cannot act on them, and did not ask which desk the copy went to. So the description above goes to the log, and the 502 body says only:

upstream request failed (request id: slpq2ff15ba63230a0d1ddec5efe)

Timeouts are the exception: the 502 body names the length that expired (upstream request timed out after 120s), because that is SLP's own configuration rather than backend topology, and it is the one detail here a caller can act on -- wait longer, or send less.

request ids

Every response carries X-Request-Id, and every log line for that request is prefixed with it. Ids match the shape gllm uses -- a four-letter prefix that never occurs in the hex body, then 12 random bytes -- so one expression finds either: (gdqz|slpq)[0-9a-f]{24}.

The prefix says who minted it, because that says how far the request got:

id	meaning
gdqz...	gllm's own id. It accepted the request; its logs have the rest.
slpq...	SLP's id, surfaced only when there is no upstream response. The request never reached the backend.

On a successful forward the upstream's X-Request-Id overwrites SLP's in the header copy, so a gdqz id reaches the client whenever gllm actually saw the request. log_requests records it as upstream=gdqz..., which is what makes an SLP log line joinable to a gllm log line for the same request.

SLP never adopts an inbound X-Request-Id. gllm ignores the request header too -- it always mints its own -- so honouring one would buy no correlation across the hop, and it would put an untrusted client string into the log. It is not dropped silently, though: the value is recorded on each request-scoped log line as client="...", and the first one seen logs a line saying it was ignored and why. Inbound values are truncated and always logged quoted, so a caller cannot forge log lines with a newline.

retry

Off by default, and copydesk editions should leave it off -- an edition with retry: true may be undesirable in many cases, since a retry on a filtered route not only adds latency; it re-runs the filter. In the case of copydesk this is a whole retrieve-and-verify pass, doubling the writer/verifier spend. On raw routes, retry: true permits a single re-dial on a transport-level error before any bytes are written. SLP never retries once the response has begun, and never retries on an HTTP status. That is the entire retry story.

Status

Roughly in dependency order:

• config load: parse backends + editions
• config compile: expand editions into a flat route table; fail closed on collisions / unknown backends / bad templates / editions with retry
• OpenAI-compatible surface: GET /v1/models from the table (?debug adds the upstream model name and route state per id, never the upstream URL); POST /v1/chat/completions (and /v1/completions, /v1/embeddings) routed by model id
• reverse proxy with FlushInterval = -1 (verbatim streaming, reasoning_content preserved)
• per-route response-header timeout via the upstream transport
• opt-in single-shot retry on pre-response transport error (raw routes only)
• opt-in stream aggregation (aggregate: true): non-streaming completions converted to upstream streaming requests and reassembled, so the route timeout bounds time-to-first-token; in-band stream errors surface as real error statuses
• request-body model extraction (small struct decode, not a full parse)
• startup route dump to logs; /healthz
• Prometheus metrics: requests and upstream latency by model id
• upstream model-name discovery: per-backend supervisor goroutine, infinite retry until found, slow drift monitoring after, explicit model: to pin; loud logs and metrics when a backend goes ambiguous

Layout

cmd/slp/          server binary
internal/config/    file schema, load, edition expansion, validation
internal/proxy/     model-id router, reverse proxy, /v1/models

Design decisions

• Dumb at runtime, by construction. The request path is read-model-id, look-up, forward. No response parsing means no response corruption: SLP's worst case is that it fails to forward, never that it mangles something mid-flight.
• Smart config compiles to a dumb table. All expansion happens once, at load. The runtime's whole world is map[modelId]upstream. Discovery is the single exception: it updates one field of one route, atomically, and cannot add, remove, or repoint a route.
• Stateless. Config file in, routes out. No database, no cache, no admin surface. (Nothing here is stateful, so nothing here is Postgres.) The discovered model names are the only runtime state, they are derived rather than authoritative, and losing them on restart costs one probe.
• Discovery degrades to the old behavior, never to a worse one. A route with no name yet forwards the body exactly as SLP did before discovery existed. Every failure mode -- backend down, catalog unparseable, several models served -- lands on pass-through or on the last known good name, and none of them can stop SLP from starting or serving.
• Separate blast radius from everything it fronts. SLP falling over must not take a backend down; it holds no state a backend depends on, and it degrades to "not running," never to "running wrong."
• Behind Caddy, not replacing it. TLS, ACME, and HTTP/2 stay upstream. SLP is only the body-aware routing Caddy structurally cannot do.

Non-goals

• Not an inference engine. Tokens belong to gllm/vLLM.
• Not a load balancer. One id, one upstream. No pools, no groups, no health-checking backends in or out of a rotation.
• Not auth or metering. No keys, no budgets, no tenancy, no admin UI. When you need per-key spend tracking across consumers you do not trust, you have outgrown SLP and want something like LiteLLM; SLP should say so rather than grow toward it.
• Not a guardrails framework. No policy engine, no request inspection beyond reading the model id.
• Not a response processor. Anything that reads or edits a response is the filter's job. SLP forwards; it does not review.

Prior art

LiteLLM is the feature-complete version of this idea and the thing to reach for once the tenancy and provider-adapter machinery earns its weight; SLP is what is left when every backend is already OpenAI-compatible and you want the router to be invisible. Caddy and nginx cannot route on a JSON body field without a custom module, which is the gap SLP fills. Bifrost sits in the same "fast Go gateway" niche with more features; SLP trades them away on purpose.

Running

slp --config slp.yaml