Energy per call: a zero is not a measurement

We wanted to show how much energy each model call used, per request, in watt-hours, all the way from the machine that served it to the screen. The plumbing took a few days. The hardest part was refusing to display a zero. This is what we learned trying to measure energy per LLM request in watt-hours on our own hardware.

Value plus source

The first decision shaped everything after it. An energy figure is never just a number. It is a number and where it came from: measured by a sensor on the machine, or estimated from a model of the hardware.

We carry both, end to end: from the model server, through the router, to the orchestrator, into the record and onto the dashboard. When figures are summed, the source is summed too. A total is measured only when every part of it was measured. One estimated part makes the whole total an estimate, and the screen says so.

1export type Energy =2  | { wh: number; source: 'measured' | 'estimated' }3  | { wh: null; source: 'missing' }4
5/** A measured zero is a sensor gap, not free inference. */6export const normalise = (e: Energy): Energy =>7  e.source === 'measured' && e.wh === 0 ? { wh: null, source: 'missing' } : e8
9export function sum(parts: Energy[]): Energy {10  const clean = parts.map(normalise)11  if (clean.some((p) => p.source === 'missing')) return { wh: null, source: 'missing' }12  const wh = clean.reduce((t, p) => t + (p.wh ?? 0), 0)13  const measured = clean.every((p) => p.source === 'measured')14  return { wh, source: measured ? 'measured' : 'estimated' }15}

A measured zero is a sensor gap

The bug that took longest to see was a zero. Some calls came back with a measured energy of exactly nothing. They looked like a triumph. They were a gap: the call was short, the sensor's sampling window missed it, and the reading was empty.

Zero is not a measurement of a working machine. So a measured zero is treated as missing. The screen shows "not measured yet" in place of a number, the figure is left out of any total that claims to be measured, and nothing downstream uses it. A blank is honest. A zero is a claim.

Streaming needs the figure after the last token

Most of our answers stream. That meant the energy figure could not travel in a response header, because headers are sent before the work is done. The machine does not know what a call used until the last token is out.

We send it as a trailing comment on the stream, after the last token. Clients that understand it read it. Clients that do not, ignore it, because it is a comment. The router passes it through and adds its own fields. The orchestrator writes it to the record with the step.

Totals that survive a restart

The dashboard shows a 24-hour total per network. The first version kept that total in memory in the gateway. Restart the gateway and the total went back to nothing, which looked exactly like the sensor bug above.

The fix was to compute the total from stored history, not from a counter. A restart now changes nothing. It costs one query, and it is correct.

What we say, and what we do not

Watts per network is a public, defensible number. We can show where it came from, what was measured and what was apportioned. A whole governed site draws 20 to 25 W. That total is measured. The split between networks on one machine is a stated convention, and the screen says which is which.

Money per token is a different conversation, with different inputs and different assumptions. We keep it out of the energy view on purpose. Mixing the two makes both harder to trust.

What to take with you

• Carry energy as value plus source. A sum is measured only when every part was.
• Treat a measured zero as a sensor gap. Show "not measured yet", never a 0.
• For streaming answers, send the figure after the last token, as a trailing comment.
• Compute running totals from stored history so a restart does not reset them.
• Keep energy and money in separate conversations.

Every step in a governed run carries its energy figure and its source. Run a scenario in the sandbox and open a record to see it.