Good Judgment, Bad Advice: a stronger controller can add cost without improving an agent

Compare the whole run

Across 21 held-out tasks and 315 trials, the reported Jev controller kept the pass rate at 73% while increasing cost by 19% relative to the baseline. The experiment holds the main agent fixed and compares several controller designs.

Good Judgment, Bad Advice: decision-model controllers ↗Controller comparison implementation ↗

Better isolated judgments can still waste work

The study links extra cost to cautious checking behavior. A controller’s score on isolated decisions therefore needs a separate comparison against completed-task outcomes and total spend. The repository includes rules and placebo comparisons to help examine that distinction.

Good Judgment, Bad Advice: decision-model controllers ↗Controller comparison implementation ↗

What you can do

1. Inspect the fixed-agent setup and cost accounting before applying the reported result to another workflow. Open reference ↗

Where the evidence stops

The findings concern this task sample, controller and agent configuration. No trial was reproduced here; the result does not rank all decision models.

Sources read. Sources checked October 21, 2026. No product workflow or benchmark was executed.

Sources & further reading