- Who it helps
- Researchers and teams comparing agent evaluations
- Availability
- Public report with a linked technical paper; some task-level and trend analyses are described as forthcoming. Original publication date was not visible in the inspected text.
Budget changes what a test can reveal
A longer run can let an agent correct mistakes; parallel attempts increase opportunities to succeed. AISI reports gains on coding, maths and cyber tasks when it raises token budgets. The same budget expansion did not help every domain: reported healthcare curves plateaued earlier.
Compare the whole setup
Record token budget, retries, agent software and cost with the result. The report supports examining a performance curve, but its findings do not promise that spending more will rescue every task or model.
What you can do
- Read the primary description and its stated limits before planning an evaluation. Open reference ↗
Where the evidence stops
Primary report read; experiments not reproduced. Illustrative figures are not measured data, and forthcoming analyses remain provisional.
Sources read. Public source text checked November 14, 2026 UTC. No third-party project, model inference or full workflow executed.
Sources & further reading
Web · paper
Compute budgets in agent evaluation ↗
Primary report separates serial and parallel budgets and describes capability curves. Some analyses remain forthcoming.
AI Security Institute · page checked 2026-11-14
More sources reviewed (1)
Web · analysis
Metrics of model ability ↗
July 24 analysis read; discusses capability measurements and cites AISI. Older analysis supplies context, without establishing current attention.
Tom Cunningham, METR · page checked 2026-11-14
Coverage note: Primary report read; experiments not reproduced. Illustrative figures are not measured data, and forthcoming analyses remain provisional.
The record over time (2)
Dated source checks and published explanations. An observation date can differ from the original release date.
- · Source recheck
AISI compares agent performance across computation budgets instead of treating one benchmark score as a capability ceiling.
- · Added to Current
AISI compares agent performance across computation budgets instead of treating one benchmark score as a capability ceiling.