feat: bill each request the energy its batches used, measured in joules #96

Merged
rcsheets merged 1 commit from feat/energy-accounting into main 2026-09-28 06:42:24 +00:00
Owner

Each request's gllm metadata already reports its compute time. This PR adds its energy in joules, reported the same two ways, and measured from the GPU board's own energy counter rather than estimated.

"gllm": {
  "batch_gpu_time_us": 1366444,
  "proportional_gpu_time_us": 1366444,
  "batch_gpu_energy_j": 160.766,
  "proportional_gpu_energy_j": 160.766,
  ...
}
  • batch_gpu_energy_j: what the GPU used during every step the request was part of, including its static draw. Those steps are shared with any other requests batched with it.
  • proportional_gpu_energy_j: the request's token-proportional share of that.
  • No meter, no fields. A device without a meter has its fields omitted, never reported as 0 J.

How it's measured

  • Sampling. NVML's board energy counter (nvmlDeviceGetTotalEnergyConsumption) advances in ~2 J steps every ~100 ms, and one read costs ~1.5 ms, both measured on this box. Reading it around each ~20 ms decode step would cost ~15% and see nothing new. Instead energy.Sample reads it every 25 ms on its own goroutine into an energy.Timeline.
  • Attribution. Each step is charged the counter's rise across its wall-clock window, interpolated linearly between readings. The counter is cumulative, so what steps are charged adds up exactly to what the card used during them.
  • The one estimate. A step that ends after the last reading is extrapolated at the latest power until the next reading arrives.
  • The GPU meter. backend.EnergyMeter is a new optional capability. The CUDA backend implements it with the NVML call, dlsym'd like the rest of NVML so it stays optional per driver, on a device handle resolved once by UUID.
  • Metrics. /metrics gains gllm_{gpu,cpu}_energy_joules_total (everything the device used while running, idle included) and …_attributed_joules_total (the part billed to steps). The gap between them is idle and host-wait energy.
  • Perfstats. The log line gains energy, average power and J per generated token.

CPU energy: slot now, source later

Options.CPUEnergy accepts any energy.Meter, which is a cumulative, non-decreasing millijoule counter. Setting it turns on batch_cpu_energy_j / proportional_cpu_energy_j and the CPU metrics.

On current kernels RAPL's energy_uj is root-only, and the image's uid 65534 can't read it at all. So the intended source is a client for a small privileged helper that reads RAPL and serves the counter, for example over a Unix socket. The helper must unwrap RAPL's wraparound (max_energy_range_uj) before handing values to gllm. AGENTS.md records this contract. Nothing sets the option yet, so CPU energy is simply omitted for now.

Validation: RTX PRO 6000, Nemotron 3 Nano 30B-A3B NVFP4, two power caps

The same three requests, served at a 200 W and then a 300 W power limit:

request 200 W: GPU time / energy / avg power 300 W: GPU time / energy / avg power change
chat (28 → 45 tokens) 1.37 s / 160.8 J / 118 W 1.27 s / 201.4 J / 159 W 7% faster, +25% energy
prefill (4314 → 1 token) 1.98 s / 225.8 J / 114 W 1.42 s / 298.6 J / 210 W 28% faster, +32% energy
decode (6 → 64 tokens) 1.60 s / 183.6 J / 114 W 1.42 s / 252.0 J / 177 W 11% faster, +37% energy
  • Decode efficiency: 2.87 J/token at 200 W, 3.94 J/token at 300 W. The higher cap buys speed at a worse energy cost per token, which is the trade-off this accounting is meant to make visible.
  • Consistency: at both caps, the three requests' batch energies sum exactly to the rise in the /metrics attributed counter (570.2 J and 751.9 J). The device total over the same spans was 571.7 J and 752.9 J; the ~1 J difference is idle time between requests.

Testing

  • energy: interpolation; exact conservation across tiled windows; extrapolation and its correction by the next reading; skipped repeats; bounded retention; the sampler's running total. All pass under -race.
  • engine:
    • A request billed from fake 150 W meters in both the backend slot and the CPU slot: it is billed exactly the attributed total, its share never exceeds its batch total, and both stay within power × wall time.
    • Unmetered means absent.
  • server: the fields are rounded to the millijoule, and omitted for an unmetered device.
  • cuda: the NVML counter reads, never goes backwards, and advances. The full CUDA suite passes (25 packages).
  • Pure-Go build: go build, go vet and go test ./... pass.
  • Docs: the README documents the fields, the metrics and the perfstats additions. AGENTS.md covers the design and the CPU helper contract.

🤖 Generated with Claude Code

Each request's `gllm` metadata already reports its compute **time**. This PR adds its **energy** in joules, reported the same two ways, and measured from the GPU board's own energy counter rather than estimated. ```json "gllm": { "batch_gpu_time_us": 1366444, "proportional_gpu_time_us": 1366444, "batch_gpu_energy_j": 160.766, "proportional_gpu_energy_j": 160.766, ... } ``` - **`batch_gpu_energy_j`**: what the GPU used during every step the request was part of, including its static draw. Those steps are shared with any other requests batched with it. - **`proportional_gpu_energy_j`**: the request's token-proportional share of that. - **No meter, no fields.** A device without a meter has its fields **omitted**, never reported as 0 J. ## How it's measured - **Sampling.** NVML's board energy counter (`nvmlDeviceGetTotalEnergyConsumption`) advances in ~2 J steps every ~100 ms, and one read costs ~1.5 ms, both measured on this box. Reading it around each ~20 ms decode step would cost ~15% and see nothing new. Instead `energy.Sample` reads it every 25 ms on its own goroutine into an `energy.Timeline`. - **Attribution.** Each step is charged the counter's rise across its wall-clock window, interpolated linearly between readings. The counter is cumulative, so what steps are charged **adds up exactly** to what the card used during them. - **The one estimate.** A step that ends after the last reading is extrapolated at the latest power until the next reading arrives. - **The GPU meter.** `backend.EnergyMeter` is a new optional capability. The CUDA backend implements it with the NVML call, dlsym'd like the rest of NVML so it stays optional per driver, on a device handle resolved once by UUID. - **Metrics.** `/metrics` gains `gllm_{gpu,cpu}_energy_joules_total` (everything the device used while running, idle included) and `…_attributed_joules_total` (the part billed to steps). The gap between them is idle and host-wait energy. - **Perfstats.** The log line gains energy, average power and J per generated token. ## CPU energy: slot now, source later `Options.CPUEnergy` accepts any `energy.Meter`, which is a cumulative, non-decreasing millijoule counter. Setting it turns on `batch_cpu_energy_j` / `proportional_cpu_energy_j` and the CPU metrics. On current kernels RAPL's `energy_uj` is root-only, and the image's uid 65534 can't read it at all. So the intended source is a client for a small **privileged helper** that reads RAPL and serves the counter, for example over a Unix socket. The helper must unwrap RAPL's wraparound (`max_energy_range_uj`) before handing values to gllm. AGENTS.md records this contract. Nothing sets the option yet, so CPU energy is simply omitted for now. ## Validation: RTX PRO 6000, Nemotron 3 Nano 30B-A3B NVFP4, two power caps The same three requests, served at a 200 W and then a 300 W power limit: | request | 200 W: GPU time / energy / avg power | 300 W: GPU time / energy / avg power | change | |---|---|---|---| | chat (28 → 45 tokens) | 1.37 s / 160.8 J / 118 W | 1.27 s / 201.4 J / 159 W | 7% faster, +25% energy | | prefill (4314 → 1 token) | 1.98 s / 225.8 J / 114 W | 1.42 s / 298.6 J / 210 W | 28% faster, +32% energy | | decode (6 → 64 tokens) | 1.60 s / 183.6 J / 114 W | 1.42 s / 252.0 J / 177 W | 11% faster, +37% energy | - **Decode efficiency:** 2.87 J/token at 200 W, 3.94 J/token at 300 W. The higher cap buys speed at a worse energy cost per token, which is the trade-off this accounting is meant to make visible. - **Consistency:** at both caps, the three requests' batch energies sum exactly to the rise in the `/metrics` attributed counter (570.2 J and 751.9 J). The device total over the same spans was 571.7 J and 752.9 J; the ~1 J difference is idle time between requests. ## Testing - **energy:** interpolation; exact conservation across tiled windows; extrapolation and its correction by the next reading; skipped repeats; bounded retention; the sampler's running total. All pass under `-race`. - **engine:** - A request billed from fake 150 W meters in both the backend slot and the CPU slot: it is billed exactly the attributed total, its share never exceeds its batch total, and both stay within power × wall time. - Unmetered means absent. - **server:** the fields are rounded to the millijoule, and omitted for an unmetered device. - **cuda:** the NVML counter reads, never goes backwards, and advances. The full CUDA suite passes (25 packages). - **Pure-Go build:** `go build`, `go vet` and `go test ./...` pass. - **Docs:** the README documents the fields, the metrics and the perfstats additions. AGENTS.md covers the design and the CPU helper contract. 🤖 Generated with [Claude Code](https://claude.com/claude-code)
feat: bill each request the energy its batches used, measured in joules
All checks were successful
ci / test_and_build (pull_request) Successful in 27s
cae0008b67
Requests already carry their compute time in the gllm metadata object:
whole-batch totals and a token-proportional share. They now carry energy
the same two ways, measured from hardware energy counters rather than
estimated:

  batch_gpu_energy_j         the GPU's energy during every step the
                             request was in, idle draw included
  proportional_gpu_energy_j  its token-proportional share

A device without a meter has its fields omitted, never reported as
0 J.

How it is measured (internal/energy):

- The counter. NVML's board energy counter advances in ~2 J steps
  every ~100 ms, and one read costs ~1.5 ms, both measured here. Reading
  it around each ~20 ms decode step would cost ~15% and see nothing, so
  energy.Sample reads it every 25 ms on its own goroutine into a
  Timeline.
- The charge. Each step is charged the counter's rise across its
  wall-clock window, interpolated linearly between readings. The
  counter is cumulative, so the charges add up exactly to what the
  device used during the steps.
- The one estimate. A step that ends after the last reading is
  extrapolated at the latest power until the next reading arrives.
- The GPU meter. backend.EnergyMeter is a new optional capability;
  the CUDA backend implements it with nvmlDeviceGetTotalEnergyConsumption,
  dlsym'd like the rest of NVML and optional per driver. The device
  handle is resolved once by UUID.

Also:

- /metrics gains gllm_{gpu,cpu}_energy_joules_total (everything the
  device used while running) and ..._attributed_joules_total (billed
  to steps); the gap is idle and host-wait energy.
- The perfstats line gains energy, average power and J per generated
  token.

CPU energy has a slot but no source yet. Options.CPUEnergy takes any
energy.Meter (a cumulative, non-decreasing millijoule counter) and lights
up the batch_cpu_energy_j/proportional_cpu_energy_j fields and CPU
metrics. RAPL's energy_uj is root-only on current kernels and
unreadable from the image's uid, so the intended source is a client for
a small privileged helper that reads RAPL, unwraps its wraparound, and
serves the counter, for example over a Unix socket. AGENTS.md records
the contract.

Validated on the RTX PRO 6000 capped at 200 W, serving Nemotron 3 Nano:

  request                     GPU time  energy    avg power  J/out token
  chat (28 -> 45 tokens)      1.37 s    160.8 J   118 W      3.57
  prefill (4314 -> 1 token)   1.98 s    225.8 J   114 W      -
  decode (6 -> 64 tokens)     1.60 s    183.6 J   114 W      2.87

The three batch energies sum to 570.2 J, which is the attributed
counter's rise. The device total rose 571.7 J over the same span, the
difference being idle time between requests. perfstats logged
gpu_energy_j=567.9 gpu_avg_power_w=113.6 for the interval that held
them.

Tests:
- energy: interpolation, exact conservation across tiled windows,
  extrapolation and its correction by the next reading, skipped
  repeats, bounded retention, and the sampler's running total.
- engine: a request billed from fake constant-power meters in both the
  backend and the CPU slot. It is billed exactly the attributed total,
  its share never exceeds its batch total, and both stay within power x
  wall time. A second test checks that unmetered means absent.
- server: the fields are rounded to the millijoule and omitted for an
  unmetered device.
- cuda: the NVML counter reads, never goes backwards, and advances. The
  full CUDA suite passes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Collaborator

Automated review by pr-reviewer v0.52.3 | Safety Check | Ministral 3 Instruct | tracking id r-ba091a-2efc86
This is an AI-generated review and may contain mistakes.

Status: ❌ Failed


Review failed. Tracking id r-ba091a-2efc86 — see logs for details.

Comment @pr-reviewer-bot retry to try again.

<!-- pr-reviewer:review --> *Automated review by [pr-reviewer](https://git.brooktrails.org/brooktrails/pr-reviewer) v0.52.3 | Safety Check | Ministral 3 Instruct | tracking id `r-ba091a-2efc86`* *This is an AI-generated review and may contain mistakes.* **Status:** ❌ Failed --- Review failed. Tracking id `r-ba091a-2efc86` — see logs for details. Comment `@pr-reviewer-bot retry` to try again.
rcsheets deleted branch feat/energy-accounting 2026-09-28 06:42:25 +00:00
Sign in to join this conversation.
No reviewers
No labels
No milestone
No project
No assignees
2 participants
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
brooktrails/gllm!96
No description provided.