Development ·
FrontierPhysics evaluates complete research artifacts with correctness gates
A new paper evaluates 56 physicist-designed investigations across 22 models, combining numerical checks with essential research-quality requirements.
Why it matters
Scientific-agent comparisons can test whether results hold up alongside the manuscript, making failures more visible before researchers rely on an automated investigation.
What to watch
Independent reruns, accessible complete evaluation artifacts and agreement between expert reviewers and automated grading.
Still uncertain. The reported experiments were not reproduced. Selected tasks, agent reviewers and configuration choices limit generalization to open-ended scientific discovery.
Impact assessment reviewed 2026-11-05.
- Who it helps
- Physics researchers and teams evaluating scientific agents.
- Availability
- November 4 preprint and a public collection of task inputs are available; a complete runnable evaluation was not verified.
What was evaluated
The authors describe 56 tasks from 42 physicists across nine domains. Across 22 models in 26 configurations, the best configuration passed 46 of 168 trials.
How a pass is decided
Numerical outputs must pass test scripts, and every essential rubric criterion must pass an agent reviewer. A well-written manuscript cannot compensate for an incorrect result. Reviewer judgments and task selection still limit what the score establishes.
FrontierPhysics checks research results and manuscripts ↗FrontierPhysics evaluation protocol ↗FrontierPhysics input collection ↗
What you can do
- Read the evaluation rules before comparing configurations. Open reference ↗
Where the evidence stops
This is a benchmark of specified research tasks, not proof of autonomous discovery. No evaluation was reproduced.
Sources read. Sources read November 5, 2026 UTC. No third-party code, model inference, account workflow or benchmark executed.
Sources & further reading
Paper · paper
FrontierPhysics checks research results and manuscripts ↗
Identifies the subject, documented capability or specific verification limit.
Bingran You and collaborators · page checked 2026-11-05
Paper · paper
FrontierPhysics evaluation protocol ↗
Identifies the subject, documented capability or specific verification limit.
Bingran You and collaborators · page checked 2026-11-05
Hugging Face · implementation
FrontierPhysics input collection ↗
Identifies the subject, documented capability or specific verification limit.
BenchFlow · page checked 2026-11-05
Coverage note: Independent attention and end-to-end execution remain unverified.
The record over time (1)
Dated source checks and published explanations. An observation date can differ from the original release date.
- · Added to Current
2026-11-04: A new paper evaluates 56 physicist-designed investigations across 22 models, combining numerical checks with essential research-quality requirements.
Assessment: Scientific-agent comparisons can test whether results hold up alongside the manuscript, making failures more visible before researchers rely on an automated investigation.