Short-task strength does not settle longer research work
In METR’s original 2024 evaluation, the tested agents outperformed humans at a two-hour budget, but humans gained more from additional time and surpassed the agents at longer budgets.
Original seven-task RE-Bench evaluation of Claude 3.5 Sonnet (New), o1-preview and human experts. The chart compares the best results from different allocations of a total time budget across independent attempts.
View original chart and methods ↗
Evaluating frontier AI R&D capabilities of language model agents against human experts · Results; paper Figure 2 and Section 4
How to interpret this chart
- This is the original 2024 evaluation, not a statement about current frontier models or the later five-task subset.
- The highest-scoring allocation can differ between agents and humans. Exact aggregate coordinates are not transcribed; the qualitative comparison is stated by METR.