Development ·
Artificial Analysis changes the task mix behind its model index
Artificial Analysis published Intelligence Index v5 with revised evaluations and sixty percent private test sets or held-out answers.
Why it matters
A widely used comparison now gives different tasks and private evaluations more weight. Teams interpreting rank changes need to distinguish the new measurement from model improvement.
What to watch
Method corrections, independently reproduced category results and whether the revised tasks predict practical performance.
Still uncertain. The composite is one weighted sample of tasks. Private components limit reproduction, and index position does not establish suitability for a particular workflow.
Impact assessment reviewed 2026-11-14.
- Who it helps
- Readers interpreting model rankings and teams selecting their own evaluations
- Availability
- October 27 announcement is public. Many component test sets or answers are private, limiting independent reproduction.
The mixture of tasks changed
The update adds or revises coding, scientific-terminal, knowledge and physics evaluations. Sixty percent of the index now uses private test sets or held-out answers. Its score reflects the chosen mixture and agent setup.
Read category results alongside the total
Nathan Lambert’s October 28 analysis argues that one overall number can obscure practical differences between models and uses. A changed ranking does not establish that a model improved on an unchanged task.
Version changes affect comparisons over time
The announcement also introduces a separate Progress Index intended to track capability on a continuous scale. Its methodology requires separate review before using it as a trend estimate.
What you can do
- Read the primary description and its stated limits before planning an evaluation. Open reference ↗
Where the evidence stops
Primary announcement and independent analysis read. No component evaluation or time-series method reproduced.
Sources read. Public source text checked November 14, 2026 UTC. No third-party project, model inference or full workflow executed.
Sources & further reading
Web · official
Artificial Analysis Intelligence Index v5 ↗
October 27 primary index announcement read. Private components limit independent replication; composite scores depend on task weights.
Artificial Analysis · page checked 2026-11-14 · Published 2026-10-27
More sources reviewed (1)
Web · analysis
Artificial Analysis v5 and benchmark selection ↗
October 28 article through line 65 inspected after main discovery. Benchmarks and named model comparisons reconciled; forecasts remain opinion.
Nathan Lambert · page checked 2026-11-14 · Published 2026-10-28
Coverage note: Primary announcement and independent analysis read. No component evaluation or time-series method reproduced.
The record over time (1)
Dated source checks and published explanations. An observation date can differ from the original release date.
- · Added to Current
2026-10-27: Artificial Analysis published Intelligence Index v5 with revised evaluations and sixty percent private test sets or held-out answers.
Assessment: A widely used comparison now gives different tasks and private evaluations more weight. Teams interpreting rank changes need to distinguish the new measurement from model improvement.