Evidence record
Research outcome prediction
1,585 later-paper idea pairs · Unadapted GPT-4.1, Table 4 baseline · 2025-06-01
Reported result · numeric
51.4%
Pairwise research-outcome prediction accuracy · percent
benchmark author reported · extraction review: agent checked
Sources
Published 2025-06-01
Evaluation setup
| split | Published-paper test after GPT-4.1 cutoff |
|---|---|
| task count | 1585 |
| task snapshot | Source paper arXiv:2506.00794v1; GPT-4.1 pre/post 1 June 2024 cutoff as applicable |
| attempts per task | 2 |
| selection rule | Predict both idea orders; count inconsistent answers incorrect |
| aggregate method | Percent correct on eligible idea pairs |
| tool access | Paper retrieval agent, before-cutoff literature only |
| filtering | Same-paper idea pairs; at least three benchmarks; ties removed |
| scaffold | Unadapted GPT-4.1 Table 4 baseline; retrieval configuration not fully specified |
| contamination concerns | Retrieval limited to papers dated before 1 June 2024 |
| comparability caveats | Table 4 prints 51.4%; Figure 3 accompanying prose separately says 51.9%. Exact configuration relationship is not reported, so the two are not averaged. |
Not reported: run count, token budget, wall clock budget, hardware, training budget, inference budget, monetary cost, internet access, evaluator version, human intervention, task exclusions.
Comparability
Not compared with other results.
Limitations
- Paper prose beside Figure 3 separately reports 51.9% for GPT-4.1; configuration relationship is not explained, so this table-specific 51.4% is not averaged with it.
- Prediction is not autonomous idea generation, experimental execution or recursive self-improvement.
- The human comparator covers a separate 45-pair NLP subset, not the full 1,585-pair test.