Flaky: TestForwardNoLeakCUDA reported a 34 MiB VRAM leak once on the RTX 3070 #126

Open
opened 2026-11-03 02:50:18 +00:00 by rcsheets · 0 comments
Owner

TestForwardNoLeakCUDA (internal/model/mistral/forward_leak_cuda_test.go) failed once on an RTX 3070 (sm_86), then passed on three immediate reruns with no code change:

--- FAIL: TestForwardNoLeakCUDA (0.05s)
    forward_leak_cuda_test.go:85: forward pass leaked 35651584 bytes of VRAM over 300 steps (~118838 B/step); free VRAM went 7768047616 -> 7732396032

Context. The run was CUDA_VISIBLE_DEVICES=<3070 uuid> go test -tags cuda ./internal/backend/cuda/ ./internal/model/mistral/ -count=1 on branch feat/decode-span-tiling (span tiling for decode attention, #124), right after make cuda. The same command passed on the same branch a few minutes earlier, before one more kernel edit. Nothing else was intentionally using the 3070; a gllm server was started on the other GPU only after the test command returned.

What the number suggests. 35,651,584 bytes is exactly 34 MiB, which is 32 MiB (STAGE_ARENA_BYTES, the per-device staging arena) plus 2 MiB. One allocation of arena size during the 300 steps would produce that. Candidates, none confirmed:

  • The test measures device-wide free memory (MemInfo), so any other process allocating on the 3070 during the window counts as a leak. A flaky failure is consistent with that.
  • A staging arena initialized lazily on a path the warm-up step did not take (for example a second device context, or gllm_stage_init running after the baseline sample).
  • The span-tiling branch adds a persistent per-device attention scratch buffer, allocated on first tiled launch. The test model's spans should be below the tiling minimum (512 positions), so it should never allocate, but this was not verified for the failing run.

To do. Make the failure reproducible (loop the test, and log nvidia-smi --query-compute-apps alongside), or make the test robust to other processes, for example by measuring the process's own allocations instead of device-wide free memory. If the scratch buffer is involved, the test should warm it up before sampling the baseline.

`TestForwardNoLeakCUDA` (`internal/model/mistral/forward_leak_cuda_test.go`) failed once on an RTX 3070 (sm_86), then passed on three immediate reruns with no code change: ``` --- FAIL: TestForwardNoLeakCUDA (0.05s) forward_leak_cuda_test.go:85: forward pass leaked 35651584 bytes of VRAM over 300 steps (~118838 B/step); free VRAM went 7768047616 -> 7732396032 ``` **Context.** The run was `CUDA_VISIBLE_DEVICES=<3070 uuid> go test -tags cuda ./internal/backend/cuda/ ./internal/model/mistral/ -count=1` on branch `feat/decode-span-tiling` (span tiling for decode attention, #124), right after `make cuda`. The same command passed on the same branch a few minutes earlier, before one more kernel edit. Nothing else was intentionally using the 3070; a gllm server was started on the other GPU only after the test command returned. **What the number suggests.** 35,651,584 bytes is exactly 34 MiB, which is 32 MiB (`STAGE_ARENA_BYTES`, the per-device staging arena) plus 2 MiB. One allocation of arena size during the 300 steps would produce that. Candidates, none confirmed: - The test measures device-wide free memory (`MemInfo`), so any other process allocating on the 3070 during the window counts as a leak. A flaky failure is consistent with that. - A staging arena initialized lazily on a path the warm-up step did not take (for example a second device context, or `gllm_stage_init` running after the baseline sample). - The span-tiling branch adds a persistent per-device attention scratch buffer, allocated on first tiled launch. The test model's spans should be below the tiling minimum (512 positions), so it should never allocate, but this was not verified for the failing run. **To do.** Make the failure reproducible (loop the test, and log `nvidia-smi --query-compute-apps` alongside), or make the test robust to other processes, for example by measuring the process's own allocations instead of device-wide free memory. If the scratch buffer is involved, the test should warm it up before sampling the baseline.
Sign in to join this conversation.
No labels
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
brooktrails/gllm#126
No description provided.