preprint · 2025-04-02
PaperBench: Evaluating AI’s Ability to Replicate AI Research
Giulio Starace, Oliver Jaffe, Dane Sherburn, James Aung, Chan Jun Shern, Leon Maksin, Rachel Dias, Evan Mays, Benjamin Kinsella, Wyatt Thompson, Johannes Heidecke, Mia Glaese, Tejal Patwardhan
Why it matters here
Paper replication receives partial rubric credit. Changing the agent scaffold helps some models while hurting another.
AI-R&D capabilityResearch autonomy
What to keep in mind
- Historical v1 extraction; not a claim about present frontier performance.