original paper

PaperBench: Evaluating AI’s Ability to Replicate AI Research (v1)

OpenAI · Published 2025-04-02

Read the original document ↗

Evidence from this source

Results and research linked to this document
TypeEvidence
ResultPaperBench · o3-mini-high / BasicAgent
Source details

Section 5.2, Table 4

ResultPaperBench · GPT-4o / BasicAgent
Source details

Section 5.2, Table 4

ResultPaperBench · Gemini 2.0 Flash / BasicAgent
Source details

Section 5.2, Table 4

ResultPaperBench · o1-high / BasicAgent
Source details

Section 5.2, Table 4

ResultPaperBench · Claude 3.5 Sonnet (New) / BasicAgent
Source details

Section 5.2, Table 4

ResultPaperBench · o3-mini-high / IterativeAgent
Source details

Section 5.3, Table 5

ResultPaperBench · Claude 3.5 Sonnet (New) / IterativeAgent
Source details

Section 5.3, Table 5

ResultPaperBench · o1-high / IterativeAgent
Source details

Section 5.3, Table 5

ResultPaperBench · o1-high / IterativeAgent, 36 h
Source details

Section 5.3, Table 5

ResultPaperBench · DeepSeek-R1 / BasicAgent
Source details

Section 5.2, Table 4, BasicAgent average Replication Scores

ResultPaperBench · o1-high / IterativeAgent, PaperBench Code-Dev
Source details

Section 2.6, Section 5.3 Table 6, PaperBench Code-Dev

ResultPaperBench · o1-high / IterativeAgent, three-paper subset
Source details

Introduction; Section 5.4 Human Baseline Performance; Figure 3 and caption

BenchmarkPaperBench
Source details

Sections 2–3 and Sections 5.2–5.3; Tables 4–5

Paper / reportPaperBench: Evaluating AI’s Ability to Replicate AI Research
Source details

Primary publication source

Document tracking details
Source ID
src-paperbench
Source type
Primary
Original URL
https://arxiv.org/html/2504.01848v1
Last updated by publisher
Not reported
Access status
available
Retrieved
2026-09-26T04:47:27Z
Document fingerprint
1704aec76a9740f8f4af9e47a9008d79aaddc17a0e16f72d3eccc39db568d27d
Based on original bytes. Used to detect changes to the source.

Availability checks

  • 2026-09-26T04:47:27Z · success: Original source inspected by Codex; extraction check, not experimental replication.