FP8 throughput: native FP8xFP8 GEMM, per-step sync, batching overlap #10

Open
opened 2026-07-02 07:46:55 +00:00 by rcsheets · 0 comments
Owner

The FP8 (E4M3) weight-quantization feature loads and serves real checkpoints correctly (validated with Mistral-Small-24B-Instruct-2501-FP8-dynamic on an RTX PRO 6000 Blackwell). The memory win is realized; throughput is not yet. Observed ~230-340 ms/token, and gpu_us ~= cpu_us on every request because the step loop blocks synchronously on the GPU.

Follow-ups, roughly in impact order:

  1. Native FP8xFP8 cuBLASLt GEMM for w8a8. Today the CUDA matMulFP8 dequantizes weights to F32 and runs the F32 GEMM for both modes; SetFP8ActivationQuant(true) only logs and falls back. A real CUDA_R_8F_E4M3 GEMM with scale attributes would put the Blackwell FP8 tensor cores in the loop.
  2. One cudaDeviceSynchronize per step, not per op. MatMul and the kernel launchers each sync (see internal/backend/cuda/), serializing the whole forward pass on the host and making cpu_us shadow gpu_us.
  3. Continuous-batching overlap. With per-op sync, concurrent requests do not actually overlap on the device.

None of these affect correctness (the CPU reference and GPU parity tests hold); they are the throughput lever after the memory win. Weight-only (W8A16) is the accurate default in the meantime.

The FP8 (E4M3) weight-quantization feature loads and serves real checkpoints correctly (validated with `Mistral-Small-24B-Instruct-2501-FP8-dynamic` on an RTX PRO 6000 Blackwell). The **memory** win is realized; **throughput** is not yet. Observed ~230-340 ms/token, and `gpu_us ~= cpu_us` on every request because the step loop blocks synchronously on the GPU. Follow-ups, roughly in impact order: 1. **Native FP8xFP8 cuBLASLt GEMM for `w8a8`.** Today the CUDA `matMulFP8` dequantizes weights to F32 and runs the F32 GEMM for both modes; `SetFP8ActivationQuant(true)` only logs and falls back. A real `CUDA_R_8F_E4M3` GEMM with scale attributes would put the Blackwell FP8 tensor cores in the loop. 2. **One `cudaDeviceSynchronize` per step, not per op.** `MatMul` and the kernel launchers each sync (see `internal/backend/cuda/`), serializing the whole forward pass on the host and making `cpu_us` shadow `gpu_us`. 3. **Continuous-batching overlap.** With per-op sync, concurrent requests do not actually overlap on the device. None of these affect correctness (the CPU reference and GPU parity tests hold); they are the throughput lever after the memory win. Weight-only (W8A16) is the accurate default in the meantime.
Sign in to join this conversation.
No labels
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
brooktrails/gllm#10
No description provided.