Long prefill: slow cancellation, and port closes before GPU memory is free #125
Loading…
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Problem
A long prefill cannot be interrupted promptly, and shutdown releases the port before it releases GPU memory.
Both will get worse as prompts grow, so they matter more once long prompts are routine.
Acceptance
Measurement conditions
gllm
00f89b1, RTX PRO 6000 Blackwell, 300 W power cap. Numbers are one replicate each. Data and scripts are on the build host in/homes/rcsheets/gllm/experiments/weight-space/(untracked):BASELINE.md,baseline.sh,prefill.sh,baseline.py. Compare only at the same power cap: start servers with--power-budget max=300W. GPU runs need the operator's approval.