OpenAI

78.05%

GPT-6 Astra

2.6 ± 0.2%

o3-mini-high / BasicAgent

4.1 ± 0.1%

GPT-4o / BasicAgent

13.2 ± 0.3%

o1-high / BasicAgent

8.5 ± 0.8%

o3-mini-high / IterativeAgent

24.4 ± 0.7%

o1-high / IterativeAgent

26.0 ± 0.3%

o1-high / IterativeAgent, 36 h

16.9 ± 1.1%

o1-preview / AIDE

8.7 ± 0.5%

GPT-4o / AIDE

138 minutes

GPT-5 / METR horizon evaluation

214 minutes

GPT-5 / METR horizon evaluation

94 minutes

o3 / METR horizon evaluation

121 minutes

o3 / METR horizon evaluation

0.191 normalized score

GPT-5.6 Sol / Codex effort aggregate

0.135 normalized score

GPT-5.6 Terra / Codex effort aggregate

0.117 normalized score

GPT-5.6 Luna / Codex effort aggregate

66.72%

GPT-6 Astra

38.12%

GPT-6 Sol

61.08%

GPT-5.6 Sol

21.83%

GPT-6 Luna

22.4%

GPT-5.6 Luna

93.80%

GPT-6 Astra

87.18%

GPT-5.6 Sol

Qualitative result

GPT-6 Astra

Qualitative result

GPT-5.6 Sol

Qualitative result

GPT-6 Astra

Qualitative result

GPT-5.6 Sol

64.20%

GPT-6 Sol

68.32%

GPT-5.6 Sol

46.62%

GPT-6 Luna

50.8%

GPT-5.6 Luna

43.4 ± 0.8%

o1-high / IterativeAgent, PaperBench Code-Dev

0.8 ± 0.5%

GPT-4o / MLAB

4.4 ± 1.4%

GPT-4o / OpenHands

173 minutes

GPT-5.1-codex-max / METR horizon evaluation

8.5 minutes

GPT-4 1106 / METR horizon evaluation

5.4 minutes

GPT-4 0314 / METR horizon evaluation

3.6 minutes

GPT-4 1106 / METR horizon evaluation

3.5 minutes

GPT-4 0314 / METR horizon evaluation

around 11.3 hours hours

GPT-5.6 Sol / METR horizon evaluation

26.6%

o1-high / IterativeAgent, three-paper subset