Long prefill: slow cancellation, and port closes before GPU memory is free #125

Open
opened 2026-11-02 22:25:39 +00:00 by rcsheets · 0 comments
Owner

Problem

A long prefill cannot be interrupted promptly, and shutdown releases the port before it releases GPU memory.

  • After a client disconnect or SIGTERM during a long prefill, the engine step ran on for 29.3 s (Granite, 70k prompt), 10.4 s (Granite, about 31k) and 7.1 s (Nemotron, about 63k). The GPU stays at full power throughout. The existing log line says a long prefill finishes its current layer first.
  • The listening port closes before that wait ends. A new server started in that window saw 25.2 GiB free instead of about 95 GiB, capped its batch to 15,448 tokens and sized its KV cache to 13,456 tokens. It logged warnings and served, then refused a 96,357-token request.

Both will get worse as prompts grow, so they matter more once long prompts are routine.

Acceptance

  • Cancellation during prefill ends the GPU work within about 1 s at 100k tokens. Chunked prefill may provide this at chunk boundaries; check it before adding anything else.
  • A supervisor has a reliable signal that GPU memory is free, such as process exit. Document it, and decide whether a server starting beside a dying predecessor should wait or fail rather than quietly undersize itself.

Measurement conditions

gllm 00f89b1, RTX PRO 6000 Blackwell, 300 W power cap. Numbers are one replicate each. Data and scripts are on the build host in /homes/rcsheets/gllm/experiments/weight-space/ (untracked): BASELINE.md, baseline.sh, prefill.sh, baseline.py. Compare only at the same power cap: start servers with --power-budget max=300W. GPU runs need the operator's approval.

## Problem A long prefill cannot be interrupted promptly, and shutdown releases the port before it releases GPU memory. - After a client disconnect or SIGTERM during a long prefill, the engine step ran on for 29.3 s (Granite, 70k prompt), 10.4 s (Granite, about 31k) and 7.1 s (Nemotron, about 63k). The GPU stays at full power throughout. The existing log line says a long prefill finishes its current layer first. - The listening port closes before that wait ends. A new server started in that window saw 25.2 GiB free instead of about 95 GiB, capped its batch to 15,448 tokens and sized its KV cache to 13,456 tokens. It logged warnings and served, then refused a 96,357-token request. Both will get worse as prompts grow, so they matter more once long prompts are routine. ## Acceptance - Cancellation during prefill ends the GPU work within about 1 s at 100k tokens. Chunked prefill may provide this at chunk boundaries; check it before adding anything else. - A supervisor has a reliable signal that GPU memory is free, such as process exit. Document it, and decide whether a server starting beside a dying predecessor should wait or fail rather than quietly undersize itself. ## Measurement conditions gllm `00f89b1`, RTX PRO 6000 Blackwell, 300 W power cap. Numbers are one replicate each. Data and scripts are on the build host in `/homes/rcsheets/gllm/experiments/weight-space/` (untracked): `BASELINE.md`, `baseline.sh`, `prefill.sh`, `baseline.py`. Compare only at the same power cap: start servers with `--power-budget max=300W`. GPU runs need the operator's approval.
Sign in to join this conversation.
No labels
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
brooktrails/gllm#125
No description provided.