- Who it helps
- Developers checking prompt, model or tool changes.
- Availability
- The repository and documented agent-regress-cli package path are public. The inspected project uses Apache 2.0 licensing.
What it accepts
The command-line tool accepts one JSON array of scores for each version. It reports a test for a difference, an estimate of its size and a confidence interval showing uncertainty under the statistical method’s assumptions. The README illustrates fifty runs per version.
What a result means
A comparison only measures the scored tasks and conditions you supplied. A p-value describes how surprising the observed difference would be under the test’s no-difference assumption; it does not measure practical importance. A stable result cannot establish that untested behavior is safe.
What you can do
- Choose a meaningful metric and fixed task set before comparing two versions. Open reference ↗
Where the evidence stops
The implementation, statistical calibration and example outputs were not executed or independently validated.
Sources read. Primary documentation read October 23, 2026 UTC. No model inference or complete product workflow was executed.
Sources & further reading
GitHub · implementation
agent-eval statistical regression tooling ↗
Read the A/B score comparison, statistical outputs and CI integration. No package was installed.
Rudrendu Paul · page checked 2026-10-23
Coverage note: Independent demonstrations and reproduced results remain unverified; the source review does not establish a tested workflow.
The record over time (1)
Dated source checks and published explanations. An observation date can differ from the original release date.
- · Added to Current
agent-eval compares repeated scores from two agent versions to help distinguish a behavior change from run-to-run variation.