Long prefill is quadratic and slow: 70k tokens takes over 10 minutes on Granite at 300 W #123

Open
opened 2026-11-02 22:25:38 +00:00 by rcsheets · 1 comment
Owner

Problem

Prefill time grows with about the square of prompt length, and is very slow at long context.

Model Prompt tokens Time to first token Prompt tok/s
Nemotron 3 Nano 6,378 2.2 s 2,909
Nemotron 3 Nano 14,542 8.2 s 1,767
Nemotron 3 Nano 30,802 33.8 s 912
Granite 4.2 8b 3,168 2.5 s 1,256
Granite 4.2 8b 7,218 12.4 s 582
Granite 4.2 8b 15,320 54.5 s 281

A 70,307-token Granite prompt had not finished at 600 s. Extrapolating the fit (not measured): Granite 70k about 19 min, Granite 96k about 36 min, Nemotron 105k about 6.5 min.

The core clock sits near 1,040 MHz at the full 300 W during prefill, against 2,730-2,835 MHz during decode, so this workload is power-limited rather than clock-limited.

Likely cause (not yet confirmed)

Exact attention is inherently quadratic, so the exponent may not change; the constant is the target. An earlier profile of a 4,314-token Nemotron prefill put about 0.47 s of 1.40 s in attention, running through the decode flash kernel (paged_attention_flash_kernel), but no profile exists at long context.

First step

Profile a Granite prefill at about 15k tokens and a Nemotron prefill at about 30k with nsys launch plus start/stop around the request. nsys profile loses CUPTI data when the Go process exits.

Acceptance

  • Profiles attached here, with a stated cost split.
  • Granite 4.2 8b prefills a 15,320-token prompt at least 5x faster at 300 W, and the same prompt's output stays within golden tolerances.
  • Revised projection for 70k and 96k prompts, measured rather than extrapolated.

Measurement conditions

gllm 00f89b1, RTX PRO 6000 Blackwell, 300 W power cap. Numbers are one replicate each. Data and scripts are on the build host in /homes/rcsheets/gllm/experiments/weight-space/ (untracked): BASELINE.md, baseline.sh, prefill.sh, baseline.py. Compare only at the same power cap: start servers with --power-budget max=300W. GPU runs need the operator's approval.

## Problem Prefill time grows with about the square of prompt length, and is very slow at long context. | Model | Prompt tokens | Time to first token | Prompt tok/s | |---|---|---|---| | Nemotron 3 Nano | 6,378 | 2.2 s | 2,909 | | Nemotron 3 Nano | 14,542 | 8.2 s | 1,767 | | Nemotron 3 Nano | 30,802 | 33.8 s | 912 | | Granite 4.2 8b | 3,168 | 2.5 s | 1,256 | | Granite 4.2 8b | 7,218 | 12.4 s | 582 | | Granite 4.2 8b | 15,320 | 54.5 s | 281 | A 70,307-token Granite prompt had not finished at 600 s. Extrapolating the fit (not measured): Granite 70k about 19 min, Granite 96k about 36 min, Nemotron 105k about 6.5 min. The core clock sits near 1,040 MHz at the full 300 W during prefill, against 2,730-2,835 MHz during decode, so this workload is power-limited rather than clock-limited. ## Likely cause (not yet confirmed) Exact attention is inherently quadratic, so the exponent may not change; the constant is the target. An earlier profile of a 4,314-token Nemotron prefill put about 0.47 s of 1.40 s in attention, running through the decode flash kernel (`paged_attention_flash_kernel`), but no profile exists at long context. ## First step Profile a Granite prefill at about 15k tokens and a Nemotron prefill at about 30k with `nsys launch` plus start/stop around the request. `nsys profile` loses CUPTI data when the Go process exits. ## Acceptance - Profiles attached here, with a stated cost split. - Granite 4.2 8b prefills a 15,320-token prompt at least 5x faster at 300 W, and the same prompt's output stays within golden tolerances. - Revised projection for 70k and 96k prompts, measured rather than extrapolated. ## Measurement conditions gllm `00f89b1`, RTX PRO 6000 Blackwell, 300 W power cap. Numbers are one replicate each. Data and scripts are on the build host in `/homes/rcsheets/gllm/experiments/weight-space/` (untracked): `BASELINE.md`, `baseline.sh`, `prefill.sh`, `baseline.py`. Compare only at the same power cap: start servers with `--power-budget max=300W`. GPU runs need the operator's approval.
Author
Owner

Measurements at a 350 W cap, with nsys kernel profiles and a vLLM 0.33.0 reference on the same prompts and the same card.

Where the time goes. paged_attention_flash_kernel is 97.0% of kernel time in a 15k-token Granite prefill (1.10 s per layer, 40 layers) and 92.7% in a 31k-token Nemotron prefill (4.53 s per attention layer, 6 layers).

Measured times, seconds to first token

Prompt gllm vLLM
Nemotron 6,378 tokens 1.86 (3 runs: 1.85-1.96) 0.33
Nemotron 30,802 30.1 (3 runs: 29.8-30.3) see below
Nemotron 87,946 236.0 see below
Granite 7,218 10.8 (3 runs) 2.03 (includes first-use kernel JIT)
Granite 15,320 47.5 (3 runs) see below
Granite 70,292 998.7 not run

Nemotron's growth from 30.8k to 87.9k has an exponent of 1.97, so prefill is still quadratic up to the longest prompt accepted today. Raising the cap from 300 W to 350 W (17%) made prefill 14-16% faster.

vLLM long-prompt numbers are not cold. Only the first prompt per model is a cold comparison. The prompts are prefixes of one another, vLLM's prefix cache was on, and its log confirms hits (31.9% on Granite), so each later request prefilled only its new tokens:

Model New tokens vLLM seconds
Nemotron 8,164 (to 14,542) 0.22
Nemotron 16,260 (to 30,802) 0.51
Nemotron 57,144 (to 87,946) 2.60
Granite 8,102 (to 15,320) 0.95
Granite 16,193 (to 31,513) 2.56

Summing the increments estimates a cold time of about 3.7 s for Nemotron 87,946 tokens against 236 s in gllm (about 65x), and 3.0 s for Granite 15,320 against 47.5 s (about 16x). These sums are estimates, not cold measurements. vLLM ran with chunked prefill at max_num_batched_tokens=8192, and its per-request time stays nearly linear with new tokens across these lengths, so the quadratic growth is not inherent to the workload. Chunking alone does not explain it, since each chunk still attends to all earlier tokens. vLLM's log shows FlashInfer prefill with an E4M3 KV cache and bfloat16 queries, which is a lead to examine.

Power during these runs was at most 358 W at 1 Hz for vLLM, and at most 364.6 W for gllm against the 350 W cap.

Measurements at a 350 W cap, with nsys kernel profiles and a vLLM 0.33.0 reference on the same prompts and the same card. **Where the time goes.** `paged_attention_flash_kernel` is 97.0% of kernel time in a 15k-token Granite prefill (1.10 s per layer, 40 layers) and 92.7% in a 31k-token Nemotron prefill (4.53 s per attention layer, 6 layers). **Measured times, seconds to first token** | Prompt | gllm | vLLM | |---|---|---| | Nemotron 6,378 tokens | 1.86 (3 runs: 1.85-1.96) | 0.33 | | Nemotron 30,802 | 30.1 (3 runs: 29.8-30.3) | see below | | Nemotron 87,946 | 236.0 | see below | | Granite 7,218 | 10.8 (3 runs) | 2.03 (includes first-use kernel JIT) | | Granite 15,320 | 47.5 (3 runs) | see below | | Granite 70,292 | 998.7 | not run | Nemotron's growth from 30.8k to 87.9k has an exponent of 1.97, so prefill is still quadratic up to the longest prompt accepted today. Raising the cap from 300 W to 350 W (17%) made prefill 14-16% faster. **vLLM long-prompt numbers are not cold.** Only the first prompt per model is a cold comparison. The prompts are prefixes of one another, vLLM's prefix cache was on, and its log confirms hits (31.9% on Granite), so each later request prefilled only its new tokens: | Model | New tokens | vLLM seconds | |---|---|---| | Nemotron | 8,164 (to 14,542) | 0.22 | | Nemotron | 16,260 (to 30,802) | 0.51 | | Nemotron | 57,144 (to 87,946) | 2.60 | | Granite | 8,102 (to 15,320) | 0.95 | | Granite | 16,193 (to 31,513) | 2.56 | Summing the increments estimates a cold time of about 3.7 s for Nemotron 87,946 tokens against 236 s in gllm (about 65x), and 3.0 s for Granite 15,320 against 47.5 s (about 16x). These sums are estimates, not cold measurements. vLLM ran with chunked prefill at `max_num_batched_tokens=8192`, and its per-request time stays nearly linear with new tokens across these lengths, so the quadratic growth is not inherent to the workload. Chunking alone does not explain it, since each chunk still attends to all earlier tokens. vLLM's log shows FlashInfer prefill with an E4M3 KV cache and bfloat16 queries, which is a lead to examine. Power during these runs was at most 358 W at 1 Hz for vLLM, and at most 364.6 W for gllm against the 350 W cap.
Sign in to join this conversation.
No labels
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
brooktrails/gllm#123
No description provided.