gllm

Kind of like vLLM, but in Go.

A minimal (but expandable) LLM inference server: OpenAI-compatible HTTP API, continuous batching, paged KV cache, CUDA backend via cgo.

Initial target: Mistral Small on an NVIDIA Blackwell GPU.

Status

Works end to end on the CPU reference backend (validated against HF transformers and mistral-common); CUDA is next. Mistral NeMo Instruct 2407 (12B) is the smallest known-compatible real checkpoint. Roughly in dependency order:

• safetensors loading (single-file and sharded checkpoints)
• config.json parsing (incl. text_config hoisting for Mistral 3/4 multimodal checkpoints)
• paged KV cache block allocator
• prefix KV caching (--prefix-cache, off by default): content-addressed, refcounted block reuse across requests sharing a prompt prefix -- suffix-only prefill, lazy LRU eviction of released blocks, preempted sequences reuse their own blocks on recompute; hit rate and block states at /metrics (gllm_prefix_cache_*). Design notes in docs/prefix-kv-caching.md
• continuous-batching scheduler (whole-prompt prefill, preemption with exact recompute when the cache fills, request cancellation; chunked prefill TODO)
• Tekken tokenizer (tekken.json, v11 pattern; hand-written linear-time splitter, goldens from mistral-common; legacy v3 files, e.g. Mistral NeMo, with synthesized specials and inline-system chat layout)
• Hugging Face tokenizer (tokenizer.json byte-level BPE, GPT-2/GPT-4 lineage; hand-written GPT-4-pattern splitter, ignore_merges fast path, added/special tokens, BOS/EOS from tokenizer_config.json; goldens from the tokenizers crate. GLM family, Llama 3, ...)
• CPU reference ops (correctness baseline for kernels; float32, float64 accumulation)
• CUDA kernels: embedding, add, rmsnorm, rope, silu_and_mul, append_kv, paged_attention (sm_120; validated op-by-op and end-to-end against the CPU reference and HF goldens)
• cuBLASLt GEMMs (row-major HF weights via a column-major op derivation)
• Mistral weight loading + forward pass (BF16 checkpoints widened to F32; validated against HF transformers on a tiny checked-in checkpoint). Multimodal nesting (model.language_model. / language_model.model., vision tower skipped), modelopt MIXED_PRECISION quant (attention FP8, MLP NVFP4), and YaRN rope scaling all load and are validated on tiny fixtures. Mistral-Medium-3.5-128B-NVFP4 runs end to end on one 96 GB GPU (generates coherent text): NVFP4 block scales stay FP8 + a global scalar (dequant in the matmul), and the auto KV-cache sizer reserves the forward's peak scratch and caps the batch to fit. Keeping embed/lm_head BF16 on device (~11 GiB) is a further optional win
• [~] GLM-5.2 (GlmMoeDsaForCausalLM): MLA + DSA sparse-attention indexer + 256-expert MoE. Host float32 CPU reference forward, validated prefill+decode against HF transformers on a tiny checkpoint. NVFP4 mixed loading (modelopt-packed experts dequantized on load, attention/shared/dense kept BF16) validated end to end on a tiny NVFP4 checkpoint. Remaining: device kernels + packed-on-device NVFP4 compute (host f32 dequant does not scale), multi-id EOS/chat template (the real ~470 GB checkpoint does not fit one GPU regardless)
• sampling (greedy, temperature/top-k/top-p, per-request seeds, EOS/length/stop-string finish)
• streaming (SSE) and the real Mistral chat template (token-level [INST]/[SYSTEM_PROMPT], goldens from mistral-common)
• logprobs: OpenAI logprobs/top_logprobs for generated tokens and vLLM's prompt_logprobs extension for prompt positions (raw log-softmax of the final logits, before temperature/top-k/top-p; per-token raw bytes; prompt path reduced host-side in bounded position chunks; non-streaming)
• [~] grammar-constrained decoding: response_format: {"type":"json_object"} guarantees valid JSON via a byte-level JSON acceptor that masks each step's logits to tokens keeping the output on a valid path (internal/grammar; forces a top-level object and EOS once complete). {"type":"json_schema"} constrains to a specific schema -- the same acceptor plus a schema cursor, so a required property cannot be dropped, an invented key never starts, and an enum value cannot be freeform text. Supported keywords are type, properties, required, enum, items, additionalProperties: false, minItems/maxItems, minimum/maximum and their exclusive forms, and the x-gllm-ordered extension (opt-in fixed emission order, taken from the required array); anything else is a 400 at request time rather than a silently unenforced constraint. Numeric bounds are decided on the digits as they arrive, not on the finished number: under minimum: 100 the digit 5 is refused where it is emitted, since no suffix could rescue it and the model would otherwise generate digits it is never allowed to stop. Still to do: $ref, oneOf/anyOf, pattern
• Prometheus metrics at /metrics (token throughput counters, weights/KV/scratch/device-memory gauges, scheduler load), plus an optional periodic perf-stats log (--log-perfstats-interval=30s) over the same registry
• per-request metadata: engine request id (X-Request-Id + a gllm response object) and per-request CPU/GPU time accounting (GPU time pending the CUDA backend)
• POST /tokenize (vLLM-compatible extension): count a prompt or messages (chat template applied) without a forward pass, so a client can size a request against the batch budget before sending; the count matches generation's prefill exactly
• KV cache sizing from free VRAM (--kv-cache auto; a unit-aware budget -- blocks/tokens/bytes -- and --max-model-len cap allocation to min(useful, affordable); host RAM sized the same way, cgroup-limit aware)
• FP8 (E4M3) weight quantization: loads pre-quantized compressed-tensors / llm-compressor checkpoints (F8_E4M3 weights + per-channel or per-tensor weight_scale), keeps them compact, and dequantizes in MatMul; --fp8-compute picks weight-only dequant (W8A16, default) or dynamic activation quant (W8A8). CPU reference validated end to end; the CUDA path dequantizes to a scratch buffer and reuses the cuBLASLt GEMM (weight-only)
• NVFP4 quantization (Blackwell 4-bit microscaling): loads pre-quantized compressed-tensors nvfp4-pack-quantized checkpoints (packed E2M1 weights two-per-byte + per-16-group E4M3 weight_scale + per-tensor FP32 weight_global_scale), folds the two-level scale into one per-group scale at load, and unpacks/dequantizes in MatMul (weight-only W4A16). CPU reference validated end to end; the CUDA path fuses the dequant into a GEMV-style kernel for batches up to nvfp4GEMVMaxRows rows (the common decode case), and falls back to unpacking+dequantizing to a scratch buffer + cuBLASLt GEMM above that (GPU-validated against the CPU reference). Still well behind vLLM's native FP4 tensor-core kernels -- see docs/perf-optimization-plan.md

Layout

cmd/gllm/                       server binary
cmd/glchat/                     reference terminal chat client (bubbletea)
internal/glchat/                instance discovery, streaming client, TUI
safetensors/                    safetensors reading library (lazy per-tensor
                                reads, sharded checkpoints, spec validation;
                                importable outside gllm)
internal/engine/                step loop; ties everything together
internal/server/                OpenAI-compatible HTTP API
internal/scheduler/             request queue, batch assembly
internal/kvcache/               paged KV cache block allocator
internal/model/                 model interface + architecture registry
internal/model/mistral/         Mistral family (GQA + RoPE + SwiGLU)
internal/model/glm/             GLM-5.2 (MLA + DSA indexer + MoE; CPU reference)
internal/backend/               compute backend interface (op set)
internal/backend/cpu/           reference backend, always available
internal/backend/cuda/          cgo CUDA backend (build tag `cuda`)
internal/backend/cuda/kernels/  .cu kernels, built by nvcc
internal/tokenizer/             Tekken / HF tokenizer loading
internal/config/                HF config.json parsing
internal/quant/                 FP8 (e4m3) + NVFP4 (e2m1) codecs + weight quantizers
internal/tensor/                dtype/shape descriptor shared by the above

Building

Pure Go (CPU backend only -- no CUDA toolkit needed):

make build          # or: go build ./...

With CUDA (requires CUDA 12.8+ for Blackwell):

make cuda                   # sm_120: GeForce RTX 50-series / RTX PRO
make cuda CUDA_ARCH=100     # sm_100: B100 / B200

Container image (pure-Go CPU build; the Kubernetes deployment lives in the brooktrails/infra repo under apps/gllm):

make image                  # tags harbor.brooktrails.org/library/gllm:<git sha>
make push
make image-cuda             # CUDA build (Dockerfile.cuda), tagged :<git sha>-cuda
make push-cuda

Running

gllm serve --model /path/to/mistral-small --addr :8000
curl localhost:8000/v1/models

From the container image

The image built by make image (and pushed by CI to harbor.brooktrails.org/brooktrails/gllm, tagged latest, the release version, and the git short sha) is the pure-Go CPU build -- no CUDA. It is built FROM scratch with the static gllm binary as the entrypoint, so everything after the image name is a gllm argument. Mount the model directory as a volume and publish the port:

docker run --rm -p 8000:8000 \
    -v /path/to/mistral-small:/model:ro \
    harbor.brooktrails.org/brooktrails/gllm:latest \
    serve --model /model --served-model-name mistral-small --addr :8000
curl localhost:8000/v1/models

Pass --served-model-name: the API model id defaults to the checkpoint directory's basename, which for a container mounted at /model is the meaningless id model -- and requests naming the model anything else are rejected with a 404.

podman run takes the same arguments. The container runs as the unprivileged user 65534 (nobody), so the model files must be readable by that uid -- world-readable files are enough; with rootless podman, add --userns=keep-id:uid=65534,gid=65534 to map your own uid onto the container user instead. On SELinux hosts use :ro,z on the volume so the mount gets a container-accessible label.

There is no shell or anything else in the CPU image; to poke around inside it, docker exec will not help -- inspect it with docker create + docker cp, or run other subcommands directly (e.g. ... gllm:latest plan --model /model --device-memory 96GiB).

That emptiness is also why the binary carries its own probe. A container healthcheck runs inside the container, where there is no curl to call /readyz with -- not in the CPU image, which has nothing at all, and not in the CUDA image either, whose base has a shell but no HTTP client. So use gllm health, which probes a running server and turns the answer into an exit status:

gllm health                          # readiness (/readyz), localhost:8000
gllm health --endpoint host:8000     # somewhere else; :8000 alone also works
gllm health --live                   # liveness (/livez) instead

As a healthcheck, in a podman quadlet or compose file:

HealthCmd=/gllm health

Readiness is the default because it answers the question a load balancer asks: it reports 503 while the model loads and once a drain begins. Liveness stays 200 for as long as the process answers HTTP at all, so restarting on it would kill instances that are merely loading or draining. Give the healthcheck a start period longer than a weight load, and prefer not to restart on failure -- a slow load must not be mistaken for a dead server.

CI also pushes a CUDA-enabled variant, tagged with a -cuda suffix (latest-cuda, <version>-cuda, <sha>-cuda) and built by Dockerfile.cuda for sm_86 and sm_120 (override CUDA_ARCHS to change that). Its runtime base is NVIDIA's CUDA runtime image, so unlike the CPU image it does have a shell. The host needs the NVIDIA driver and nvidia-container-toolkit; hand the GPU to the container with --gpus all (docker) or --device nvidia.com/gpu=all (podman, CDI):

docker run --rm --gpus all -p 8000:8000 \
    -v /path/to/mistral-small:/model:ro \
    harbor.brooktrails.org/brooktrails/gllm:latest-cuda \
    serve --model /model --served-model-name mistral-small --addr :8000

A quantized checkpoint is detected from its tensor dtypes and config.json (quantization_config); no flag is needed to serve one. FP8 and NVFP4 are both recognized automatically. --fp8-compute chooses how FP8 weights are used: weight-only (dequantize weights, keep activations full precision; the default) or w8a8 (also quantize activations, FP8 x FP8 -- reference on CPU; the CUDA backend falls back to weight-only). NVFP4 checkpoints serve weight-only (W4A16): the packed 4-bit weights are unpacked and dequantized in MatMul, with no flag to set. For which FP8 checkpoints to try and how, see docs/fp8-checkpoints.md.

To decide where to run a model before committing a GPU, gllm plan estimates whether it fits on a device of a given size and the KV headroom it would leave, without a GPU and without holding the weights resident:

gllm plan --model /path/to/model --device-memory 96GiB
gllm plan --model /path/to/model --device-memory 96GiB --json   # for an orchestrator

It measures the weight footprint by running the real loader against a sizing backend that tallies allocations instead of reserving them (so quantized compaction, checkpoint nesting, and multimodal skips are exact, not guessed) and splits the remaining memory with the same auto:fill math serve uses. The weights must be readable, but only one tensor is held at a time. --max-seqs, --max-model-len, --max-batch-tokens, and --kv-block-size mirror serve so the plan reflects how the model would actually run. Architectures whose weights are not device-resident (GLM's host-float32 reference forward) are reported as an error rather than a bogus footprint.

Every response carries an X-Request-Id header and a gllm metadata object (the request id, the batches it ran in, and its CPU/GPU time). Disable the body object with serve --request-metadata=false or a per-request "gllm_metadata": false; the header is always sent.

Prometheus metrics are served at GET /metrics (token throughput counters, weights/KV/scratch/device-memory gauges, and scheduler load). For a log-only view, serve --log-perfstats-interval=30s logs a performance summary every 30s from the same metrics -- prompt tokens ingested and tokens generated in the interval (with per-second rates), a memory breakdown (weights, KV cache, transient scratch, device free/total), and the scheduler load; omit the flag (or set 0) to disable it.

Two gllm-internal management endpoints (not part of the OpenAI API) support orchestration: GET /v1/internal/status returns a JSON snapshot -- the served model, resolved capacity (max seqs / model len / batch tokens, KV cache size), device footprint and free/total memory, live running/waiting load, build version, whether HTTP drain is enabled, and (when set) the request limit and completed count -- so an orchestrator can route requests and make model-placement decisions. POST /v1/internal/drain begins the same graceful drain as a SIGTERM (stop accepting new requests, finish in-flight, then exit), returning 202 immediately and idempotent, so an instance can be retired over the wire without shell or signal access to the box. The drain endpoint is gated on a shared secret: it is disabled unless serve --drain-secret=<secret> is set, and callers must then present that secret verbatim in the X-Drain-Secret header (SIGTERM drains regardless).

Both endpoints answer from the moment the port binds, before the (minutes-long) model load finishes: during the load the status snapshot is {"state": "loading", "model_dir": ..., "elapsed_seconds": ...} so an orchestrator can tell "starting" from "dead", the drain endpoint aborts the load and exits (same secret gating), and every other endpoint answers 503 with a Retry-After hint instead of hanging. Once the load completes the status payload reports "state": "serving" (then "draining" after a drain is triggered).

Two probe endpoints follow the k8s liveness/readiness split. GET /livez is liveness: 200 whenever the process is up and answering HTTP, including during the load and a graceful drain -- point restart-on-failure supervision here, since restarting an instance for loading or draining defeats both. GET /readyz is readiness: 200 only while the instance should receive traffic -- 503 during the load and 503 again once a drain has been triggered, so a load balancer stops routing new requests while in-flight ones finish. GET /healthz is a deprecated, temporary alias of /readyz from before the split; migrate probes to /livez + /readyz. Each IP still probing the alias is warned about once in the server log, so the log names every prober left to migrate.

At --log-level debug each request logs an engine: request received line when it is admitted (paired with the INFO engine: request complete line by request_id), so tailing the log shows what is in flight. Add serve --debug-with-prompt-preview to include the text of the first few prompt tokens as a prompt_preview field, with control tokens shown by name (e.g. <s>[INST]hi there) rather than dropped (off by default -- it logs prompt content); --debug-prompt-preview-size=N sets how many tokens.

To exercise the whole stack on a CPU-only machine -- including building a tiny servable checkpoint to curl -- see docs/end-to-end-cpu.md.

Chatting with a server

glchat is a reference terminal client, built by make build alongside the server (or on its own with make glchat):

glchat                          # find a local server and chat with it
glchat --endpoint host:9000     # talk to a specific one

With no flags it probes the port range gllm serve binds (:8000 plus the 20 it walks forward to when that is taken), asking each for /v1/internal/status. That confirms an instance is live and past its model load, and reports what it is serving -- so the model name never has to be typed. One instance connects straight through; several open a picker ordered by load. Replies stream token by token; ctrl+c interrupts a generation, and again quits.

gllm chat runs the same client. It is a wrapper that execs the glchat binary rather than a subcommand proper, so the server binary does not link the TUI's dependencies; it is only offered when that binary is present next to gllm or on PATH.