feat: bill each request the energy its batches used, measured in joules #96
Loading…
Reference in a new issue
No description provided.
Delete branch "feat/energy-accounting"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Each request's
gllmmetadata already reports its compute time. This PR adds its energy in joules, reported the same two ways, and measured from the GPU board's own energy counter rather than estimated.batch_gpu_energy_j: what the GPU used during every step the request was part of, including its static draw. Those steps are shared with any other requests batched with it.proportional_gpu_energy_j: the request's token-proportional share of that.How it's measured
nvmlDeviceGetTotalEnergyConsumption) advances in ~2 J steps every ~100 ms, and one read costs ~1.5 ms, both measured on this box. Reading it around each ~20 ms decode step would cost ~15% and see nothing new. Insteadenergy.Samplereads it every 25 ms on its own goroutine into anenergy.Timeline.backend.EnergyMeteris a new optional capability. The CUDA backend implements it with the NVML call, dlsym'd like the rest of NVML so it stays optional per driver, on a device handle resolved once by UUID./metricsgainsgllm_{gpu,cpu}_energy_joules_total(everything the device used while running, idle included) and…_attributed_joules_total(the part billed to steps). The gap between them is idle and host-wait energy.CPU energy: slot now, source later
Options.CPUEnergyaccepts anyenergy.Meter, which is a cumulative, non-decreasing millijoule counter. Setting it turns onbatch_cpu_energy_j/proportional_cpu_energy_jand the CPU metrics.On current kernels RAPL's
energy_ujis root-only, and the image's uid 65534 can't read it at all. So the intended source is a client for a small privileged helper that reads RAPL and serves the counter, for example over a Unix socket. The helper must unwrap RAPL's wraparound (max_energy_range_uj) before handing values to gllm. AGENTS.md records this contract. Nothing sets the option yet, so CPU energy is simply omitted for now.Validation: RTX PRO 6000, Nemotron 3 Nano 30B-A3B NVFP4, two power caps
The same three requests, served at a 200 W and then a 300 W power limit:
/metricsattributed counter (570.2 J and 751.9 J). The device total over the same spans was 571.7 J and 752.9 J; the ~1 J difference is idle time between requests.Testing
-race.go build,go vetandgo test ./...pass.🤖 Generated with Claude Code
Requests already carry their compute time in the gllm metadata object: whole-batch totals and a token-proportional share. They now carry energy the same two ways, measured from hardware energy counters rather than estimated: batch_gpu_energy_j the GPU's energy during every step the request was in, idle draw included proportional_gpu_energy_j its token-proportional share A device without a meter has its fields omitted, never reported as 0 J. How it is measured (internal/energy): - The counter. NVML's board energy counter advances in ~2 J steps every ~100 ms, and one read costs ~1.5 ms, both measured here. Reading it around each ~20 ms decode step would cost ~15% and see nothing, so energy.Sample reads it every 25 ms on its own goroutine into a Timeline. - The charge. Each step is charged the counter's rise across its wall-clock window, interpolated linearly between readings. The counter is cumulative, so the charges add up exactly to what the device used during the steps. - The one estimate. A step that ends after the last reading is extrapolated at the latest power until the next reading arrives. - The GPU meter. backend.EnergyMeter is a new optional capability; the CUDA backend implements it with nvmlDeviceGetTotalEnergyConsumption, dlsym'd like the rest of NVML and optional per driver. The device handle is resolved once by UUID. Also: - /metrics gains gllm_{gpu,cpu}_energy_joules_total (everything the device used while running) and ..._attributed_joules_total (billed to steps); the gap is idle and host-wait energy. - The perfstats line gains energy, average power and J per generated token. CPU energy has a slot but no source yet. Options.CPUEnergy takes any energy.Meter (a cumulative, non-decreasing millijoule counter) and lights up the batch_cpu_energy_j/proportional_cpu_energy_j fields and CPU metrics. RAPL's energy_uj is root-only on current kernels and unreadable from the image's uid, so the intended source is a client for a small privileged helper that reads RAPL, unwraps its wraparound, and serves the counter, for example over a Unix socket. AGENTS.md records the contract. Validated on the RTX PRO 6000 capped at 200 W, serving Nemotron 3 Nano: request GPU time energy avg power J/out token chat (28 -> 45 tokens) 1.37 s 160.8 J 118 W 3.57 prefill (4314 -> 1 token) 1.98 s 225.8 J 114 W - decode (6 -> 64 tokens) 1.60 s 183.6 J 114 W 2.87 The three batch energies sum to 570.2 J, which is the attributed counter's rise. The device total rose 571.7 J over the same span, the difference being idle time between requests. perfstats logged gpu_energy_j=567.9 gpu_avg_power_w=113.6 for the interval that held them. Tests: - energy: interpolation, exact conservation across tiled windows, extrapolation and its correction by the next reading, skipped repeats, bounded retention, and the sampler's running total. - engine: a request billed from fake constant-power meters in both the backend and the CPU slot. It is billed exactly the attributed total, its share never exceeds its batch total, and both stay within power x wall time. A second test checks that unmetered means absent. - server: the fields are rounded to the millijoule and omitted for an unmetered device. - cuda: the NVML counter reads, never goes backwards, and advances. The full CUDA suite passes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>Automated review by pr-reviewer v0.52.3 | Safety Check | Ministral 3 Instruct | tracking id
r-ba091a-2efc86This is an AI-generated review and may contain mistakes.
Status: ❌ Failed
Review failed. Tracking id
r-ba091a-2efc86— see logs for details.Comment
@pr-reviewer-bot retryto try again.