benchmark
CoBench
Diagnose historical internal R&D failures from infrastructure snapshots.
AI-R&D capability
Reported result55.8%2.1 · Reported scoreClaude Opus 5.5 · 2026-09-22Other settings are available.See all results
Estimated requirement85%
Directory
Benchmarks & indicators
Measures of AI research and self-improvement ability.
benchmark
Internal Research Debugging Evaluation
Resolve real bugs in internal research experiments.
AI-R&D capability
Reported result75.52%GPT-6.1 Sol card snapshot, September 29, 2026 · Mean rubric rewardGPT-6.1 Sol · 2026-09-29See all results
benchmark
KernelGen 1P
Optimize correct kernels for OpenAI first-party hardware.
AI-system improvement
Reported result60.44%GPT-6.1 Sol card snapshot, September 29, 2026 · Mean kernel optimization rewardGPT-6.1 Sol · 2026-09-29See all results
benchmark
NanoGPT
Reduce small-model training time to a target validation objective.
AI-system improvement
Reported resultResults published as a chartExact point values are not printed.GPT-6 Astra · 2026-09-03Evidence record · Original source ↗
Human reference0.7238normalized scoreVersion Astra card snapshot, September 2026
benchmark
PostTrainBench Lite
Improve a pretrained model within a five-hour GPU budget.
AI-system improvement
Reported resultResults published as a chartExact point values are not printed.GPT-6 Astra · 2026-09-03Evidence record · Original source ↗
Baseline reference1normalized scoreVersion Astra card snapshot, September 2026
benchmark
GRB internal research engineering
End-to-end research engineering in an internal coding scaffold.
AI-R&D capability
Reported result27%Average pass@1Gemini 3.7 Flash / GRB scaffold · 2026-08 · 2 hours per taskOther settings are available.See all results
Policy reference90%
benchmark
PaperBench
Replicate research papers against author-developed rubrics.
AI-R&D capability
Reported result21.0 ± 0.8%Standard errorOriginal paper v1 · Mean replication scoreClaude 3.5 Sonnet (New) / BasicAgent · 2025-04-02 · 12 hoursOther settings are available.See all results
benchmark
MLE-bench
ML competition engineering under bounded compute.
AI-R&D capability
Reported result93.80%Revised, Astra card snapshot · Mean percentile against reference solutionsGPT-6 Astra · 2026-09-03 · Max reasoningSee all results
benchmark
RE-Bench
Research engineering environments with expert baselines.
AI-R&D capability
Reported result1.27normalized scoreGoogle model-card assessment, February 2026 · Human-normalised average scoreGemini 3.1 Pro (Deep Think) · 2026-02-19See all results
Baseline reference1normalized score
benchmark
Task-completion time horizons
Human-task duration associated with 50% agent success.
Research autonomy
Reported resultaround 11.3 hoursConfidence interval: 95% CI: 5–40 hoursCheating counted as failure; METR considers this estimate not reliable.TH1.1, GPT-5.6 Sol June 2026 assessment · 50% task-completion horizonGPT-5.6 Sol / METR horizon evaluation · 2026-06-26See all results
benchmark
AI4AI-Bench
Rewrite training algorithms in frozen research repositories.
AI-system improvement
Reported result0.250normalized scoreMean normalized scoreClaude Opus 5 / Claude Code effort aggregate · 2026-08-20 · 4 hours explorationOther settings are available.See all results
Baseline reference0.1normalized score
operational metric
Anthropic R&D Automation Index
Model-rated automation across a fixed basket of R&D work.
Observed R&D automation
operational metric
Experienced developer productivity
Randomized access to AI tools on real repository issues.
Observed R&D automation
Reported result-18%Confidence interval: -38% to +9% · Source-reported confidence interval; level not specified in summaryLate-2025 study update · AI-assisted task-time changeMETR · 2026-02-24 · Returning original-study participantsOther settings are available.See all results
Also reportedMETR reports that selection effects make the later experiment unreliable for estimating current productivity gains. · 2026-02-24Evidence record · Original source ↗
operational metric
AlphaEvolve training optimization
Reported deployment of AI-discovered training kernels.
AI-system improvementObserved R&D automation
Reported result1%Gemini training-time reductionAlphaEvolve / Gemini ensemble · 2025-05-14See all results
benchmark
Research outcome prediction
Tests whether a specialized AI system can predict which of two research ideas will perform better before the experiment.
AI-R&D capability
Reported result77.0%1,585 later-paper idea pairs · Pairwise research-outcome prediction accuracyFine-tuned GPT-4.1 plus paper retrieval · 2025-06-01 · Published-paper test after GPT-4.1 cutoffOther settings are available.See all results
reported capability index
Automated weak-to-strong research study
Tests whether AI research agents can find better ways to train a stronger model using guidance from a weaker model.
AI-R&D capabilityAI-system improvement
Reported result0.97 PGRAgents received evaluation scores throughout the search; this was not an untouched final test.Performance gap recoveredNine Claude Opus 4.6 research agents · 2026-04 · Chat preference ID+OOD held-out splitsOther settings are available.See all results
Human reference0.23 PGR
benchmarkBenchmark status: active
PostTrainBench
Tests how well AI agents improve small AI models, with ten hours and one H100 GPU per training run.
AI-R&D capabilityAI-system improvementEvaluation integrity
Reported result45.3%PostTrainBench v1.1 / Google OpenCode evaluation, October 2026 · Weighted downstream benchmark scoreGemini 4 Argon / Google OpenCode evaluation · 2026-10 · 10 hoursSee all results
benchmarkBenchmark status: active
TasteVal
Measures how efficiently models choose experiments and interpret results on eight fixed AI R&D problems, with a shared coding agent and human expert reference.
AI-R&D capabilityAI-system improvementEvaluation integrity
Reported result1.72×Hierarchical bootstrap: 95% CI 0.75–3.70×Scored on seven tasks; one refused task was excluded as missing data, not failure or zero.Experimental compute multiplierFable 5.1 (max) / TasteVal · 2026-10-05 · Train and validation visible to ResearcherOther settings are available.See all results
Human reference1 ×
benchmarkBenchmark status: active
InnovationEval
Tests whether an AI agent can develop a post-training method approaching a recent human innovation, from ideas through implementation and experiments.
AI-R&D capabilityAI-system improvementEvaluation integrity
Reported result2%Estimated seed/rerun confidence interval: 90% CI 0–10%; training noise, one agent attemptEpoch adjusted these scores for run selection and method-scope judgments. One main research attempt was evaluated per model; no paper recall was detected, but contamination is not ruled out.Epoch-corrected normalized in-scope scoreClaude Fable 5 · InnovationEval · 2026-10-07 · Visible short-answer test setsSee all results
Baseline reference100%
No matching measures.
Try removing a filter or using a broader search.