FP8 throughput: native FP8xFP8 GEMM, per-step sync, batching overlap #10
Loading…
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
The FP8 (E4M3) weight-quantization feature loads and serves real checkpoints correctly (validated with
Mistral-Small-24B-Instruct-2501-FP8-dynamicon an RTX PRO 6000 Blackwell). The memory win is realized; throughput is not yet. Observed ~230-340 ms/token, andgpu_us ~= cpu_uson every request because the step loop blocks synchronously on the GPU.Follow-ups, roughly in impact order:
w8a8. Today the CUDAmatMulFP8dequantizes weights to F32 and runs the F32 GEMM for both modes;SetFP8ActivationQuant(true)only logs and falls back. A realCUDA_R_8F_E4M3GEMM with scale attributes would put the Blackwell FP8 tensor cores in the loop.cudaDeviceSynchronizeper step, not per op.MatMuland the kernel launchers each sync (seeinternal/backend/cuda/), serializing the whole forward pass on the host and makingcpu_usshadowgpu_us.None of these affect correctness (the CPU reference and GPU parity tests hold); they are the throughput lever after the memory win. Weight-only (W8A16) is the accurate default in the meantime.