- Who it helps
- Developers and researchers evaluating routing, checks and other decisions inside agents.
- Availability
- The October 20 paper and repository are public. Code uses Apache-2.0 and generated core items use CC BY 4.0; reference tasks retain their original licenses. Model inference can incur provider costs.
A stronger check than one correct answer
A contrast pair changes one decisive fact while keeping the surrounding case similar. Pair accuracy requires getting every member right. The authors report substantial differences between models with similar aggregate scores.
Record the exact dataset version
The paper uses 7,800 generated core items and 32,233 reference items in release 1.3. The downloadable release 1.1 includes additional reference items; the repository documents deriving 1.3 and checking hashes. Mixing those versions would change the comparison.
DecisionPivot: a diagnostic benchmark for decision models ↗DecisionPivot code, versions and scoring ↗
Confidence has to match errors
The scoring tools measure calibration, meaning how closely confidence matches correctness. Their automation-at-5%-error measure concerns the evaluated sample and its ordering by confidence. It does not establish a deployment error guarantee.
What you can do
- Read the task cards, version table and licensing before deciding which tests represent your agent’s decisions. Open reference ↗
Where the evidence stops
The paper and repository were read. No generator, solver or model evaluation was executed, and results remain author-reported.
Sources read. Sources checked October 21, 2026. No product workflow or benchmark was executed.
Sources & further reading
Paper · paper
DecisionPivot: a diagnostic benchmark for decision models ↗
Explains the paired-case test and the dataset used in the paper.
Xialie Zhuang and coauthors · page checked 2026-10-21 · Published 2026-10-20
GitHub · implementation
DecisionPivot code, versions and scoring ↗
Provides version differences, licensing and scoring code for inspecting or reproducing the benchmark.
StartLux Labs · page checked 2026-10-21
Coverage note: Independent hands-on testing and cross-platform discussion are incomplete. No missing links, creator identities or engagement counts are inferred.
The record over time (2)
Dated source checks and published explanations. An observation date can differ from the original release date.
- · Source recheck
DecisionPivot evaluates models used to make decisions inside AI systems, using paired cases whose correct answer changes when one crucial detail changes.
- · Added to Current
DecisionPivot evaluates models used to make decisions inside AI systems, using paired cases whose correct answer changes when one crucial detail changes.