The planner said done, so the orchestrator said success

We found a turn in our logs where every step failed and the status said success. Worse, the final answer was confident and specific. If you have seen an LLM agent report success when all tool calls failed, the cause is probably the one we had: success came from the model, not from the results.

How a failed turn becomes a success

The setup is common. A planner model breaks an objective into steps. An orchestrator runs each step by calling an agent or a tool. When the planner thinks the work is finished, it sets a done flag. An aggregator model then writes the answer from the step results.

In the turn we looked at, from a demo with several agents, each step came back as a failure. The planner saw the failures, decided there was nothing more to try, and set done. The orchestrator read done as "finished well" and marked the turn a success. The aggregator was handed an empty set of results and a success status, and did what language models do with a blank page and a confident frame. It wrote something plausible.

Nobody lied. Every component trusted the one upstream of it. The bug is in where the decision lived.

The outcome is a function of the step results

The fix is a rule, and the rule is simple: the outcome of a turn is computed in code from the step results. A model's stop or done field is advice about whether to keep planning. It is never the verdict.

1type StepResult = { ok: boolean; error?: string }2
3export function turnOutcome(steps: StepResult[]): 'success' | 'partial' | 'failed' {4  if (steps.length === 0) return 'failed'5  const ok = steps.filter((s) => s.ok).length6  if (ok === steps.length) return 'success'7  return ok === 0 ? 'failed' : 'partial'8}

When the outcome is failed, the aggregator is not asked to write an answer. It is asked to explain what failed, from the errors, in plain words. That is a different prompt with a different job, and it is much harder for it to invent a result.

We apply this on every runtime we support, not only the one where we caught it. A rule that lives in one code path is a bug waiting in the others.

Deciding what counts as a failure

Once code owns the verdict, the next question is what a failure looks like. This was less obvious than it sounds.

A transport error is a failure. A non-success status code is a failure. But some agents answer with a normal status and a body that says, in effect, success false, with no error key and no message. We had been reading that as a success because nothing looked like an error. Now a body that reports it did not succeed is a failure, whatever the status code says.

Repeated failures needed a rule too. When a step fails, the planner gets a chance to re-plan. That is useful the first time. When the same call fails with the same error twice, re-planning is not finding a new route. It is spinning. We bound it in code: the same call with the same error twice is a stop. The turn ends, the outcome is computed, and a human can see why.

Different models, different control fields

The last lesson came from swapping models. A larger planner tended to emit a done flag. A smaller one sometimes emitted a stop reason instead, or neither, and simply returned an empty plan. Any rule that depends on one model's control field is a latent bug the day you change models.

So we treat every model control field as a hint, normalise the hints we know about, and never let any of them decide the outcome. The orchestrator ends a turn when there is nothing left to run, when a bound is hit, or when a step is refused. Those are facts the orchestrator owns.

This also changed how we test. We run the same objective through each planner size and check the outcome against the step results, not against the text of the answer. A small model is better at finding these gaps than a large one, because it guesses less.

What to take with you

• Compute the outcome of a turn from step results, in code. A model's done flag is advice.
• When every step failed, do not ask a model to write the answer. Ask it to explain the failure.
• A body that says it did not succeed is a failure, with or without a status code.
• The same call with the same error twice is a stop, not a re-plan.
• Do not depend on one model's control fields. Test with a small model; it hides less.

Every step's result, including every failure, lands on the record with its reason. Watch a run fail honestly in the sandbox.