FrontierPhysics: Evaluating Agents for End-to-End Physics Research

FrontierPhysics is a 56-task, expert-built end-to-end physics research benchmark that grades deterministic artifacts and manuscripts, finding that frontier agents rarely exceed a 27.4% pass rate and mainly fail from weak rigor and limited expert intuition.

Existing physics benchmarks evaluate fixed questions, isolated sub-tasks, or paper reproduction with given solutions, and rarely check research quality alongside correctness. Many closed-ended benchmarks saturate, while full-paper reproductions specify research questions and solution structures in advance. The paper argues that no existing benchmark is fully agentic, built by physicists from their own research, graded on both paper quality and result correctness, and supported by manual task review and failure-mode analysis.

FrontierPhysics contains 56 tasks made by 42 physicists from their own research across nine domains in theoretical, computational, and experimental physics. Each task gives a high-level PI-like prompt and may provide input data or supporting files, but no explicit methodology or solution pathways. Agents must review literature, develop a research plan, conduct research and experiments, and produce deterministic artifacts plus a paper.pdf manuscript. Evaluation combines deterministic verifiers for structured outputs with an Agent-as-judge reviewer using global and task-specific rubrics. Rubric criteria are split into binary blockers, which must all pass, and weighted non-blockers scored 0, 1, or 2, with the final score gated by the verifier and blockers as in Eq. 1.

• Overall pass rates are below 30% for all tested agents, and Claude Opus 5.5 has the highest pass rate at 27.4%. §3.2
• No configuration passes more than 46 of its 168 rollouts. §Abstract
• Correcting eight FALSE and one RRMIS outcomes changes GPT-6 Astra xhigh pass rate from 14.9% (25/168) to 20.2% (34/168). §3.4
• Agents mainly fail from insufficiently rigorous reasoning at 48%, limited physical intuition at 34%, literature synthesis at 27%, and instruction following at 26%, rather than mostly physics mistakes. §3.4
• In a rubric sanity check on 33 tasks with author-provided reference materials, the authors' own materials score 96.2% on average under the corresponding rubrics. §3.3
• Pass rate generally rises with cost, but for some Claude models it decreases as cost increases in the order Opus 5.5, Fable 5.1, and Opus 5, while GPT models are generally more cost-efficient. §3.2

• The failure-mode analysis focuses only on GPT-6 Astra because manually reading trajectories with subagents is challenging; other models or model-harness pairs may fail differently, and only GPT-6 Astra false negatives are accurately tagged and corrected.
• Verifier and rubric defects may remain; nine of the 168 rollouts incorrectly failed, and finding false negatives requires both domain expertise and time.
• With few tasks per domain, differences between physics domains are uncertain.
• Regrading with Claude Opus 5.5 as reviewer gives the same pass/fail decision for almost all GPT-6 Astra rollouts, but the Opus 5.5 reviewer is more lenient on rollouts from its own model family.

• FrontierPhysics contains 56 tasks made by 42 physicists from their own research across nine physics domains; each task author has 420 citations on average. §1
• Each task requires deterministic artifacts and a paper.pdf manuscript, and some tasks provide input data or supporting files, but no explicit methodology or solution pathways are specified. §2.1
• Evaluation combines deterministic verifiers with an Agent-as-judge reviewer because many rubric criteria must be checked against trajectories, code, and output files rather than paper.pdf alone. §2.3
• Blockers are binary gates essential to scientific validity, non-blockers are scored 0 to 2 with weights from 1 to 10, and high non-blocker scores cannot compensate for a failed verifier or blocker. §2.3
• The experiments evaluate 22 frontier models in four agent harnesses; each model attempts all 56 tasks three times at its highest available inference effort, and four GPT models are also run at xhigh. §3
• All rollouts have a uniform 10-hour timeout cap, and model-caused endings stay in the denominator while infrastructure-caused endings are re-run. §3.1
• Manual review finds frontier agents' main weaknesses are insufficiently rigorous reasoning and limited expert intuition, rather than gaps in physics knowledge or execution errors. §Abstract
• The conclusion argues that capable physics research agents should go beyond specified tasks toward independent exploration from problem definition to novel findings with only high-level guidance, requiring more open-ended tasks and flexible evaluation. §4