Flaky: TestForwardNoLeakCUDA reported a 34 MiB VRAM leak once on the RTX 3070 #126
Loading…
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
TestForwardNoLeakCUDA(internal/model/mistral/forward_leak_cuda_test.go) failed once on an RTX 3070 (sm_86), then passed on three immediate reruns with no code change:Context. The run was
CUDA_VISIBLE_DEVICES=<3070 uuid> go test -tags cuda ./internal/backend/cuda/ ./internal/model/mistral/ -count=1on branchfeat/decode-span-tiling(span tiling for decode attention, #124), right aftermake cuda. The same command passed on the same branch a few minutes earlier, before one more kernel edit. Nothing else was intentionally using the 3070; a gllm server was started on the other GPU only after the test command returned.What the number suggests. 35,651,584 bytes is exactly 34 MiB, which is 32 MiB (
STAGE_ARENA_BYTES, the per-device staging arena) plus 2 MiB. One allocation of arena size during the 300 steps would produce that. Candidates, none confirmed:MemInfo), so any other process allocating on the 3070 during the window counts as a leak. A flaky failure is consistent with that.gllm_stage_initrunning after the baseline sample).To do. Make the failure reproducible (loop the test, and log
nvidia-smi --query-compute-appsalongside), or make the test robust to other processes, for example by measuring the process's own allocations instead of device-wide free memory. If the scratch buffer is involved, the test should warm it up before sampling the baseline.