report2026-11-09
Anthropic
Anthropic's November 9 system card updates its account of AI research and development capability for Claude Fable 5.2 and Mythos 5.2. It reports a revised capability-index fit that favors sustained faster growth over the previously preferred one-time jump, while retaining uncertainty about AI's causal contribution and judging that Mythos 5.2 does not fully substitute for its research staff. The card's quoted METR testing finds stronger verifiable-task performance on an earlier, more capable snapshot, with mixed evidence for open-ended research judgment. These are qualified evaluation and methodology updates, not demonstrated autonomous recursive improvement.
AI-R&D capabilityResearch autonomyEvaluation integrity
report2026-11-05
Tom Cunningham, Nate Rush, Parker Whitfill, Hjalmar Wijk
METR reports an August–October 2026 assessment of AI R&D acceleration at Anthropic using internal capability histories, research-input data and staff interviews. It judges that progress on well-specified tasks has accelerated and that AI assistance likely contributes substantially, while distinguishing observed benchmark trends from uncertain causal attribution. Highly autonomous contributions are not yet described as a major source of overall progress. The public account exposes important measurement, access and redaction limits rather than independently demonstrating a recursive improvement loop.
AI-R&D capabilityEvaluation integrity
report2026-11-05
Tom Cunningham, Nate Rush, Parker Whitfill, Hjalmar Wijk
METR's November 2026 review distinguishes Anthropic's AI R&D early-warning threshold from full substitution of researchers and from direct catastrophic risk. For models covered through July 15, METR broadly agrees with the low direct-risk and non-substitution conclusions but considers crossing the acceleration warning threshold plausible and the public report's confidence overstated. It identifies omitted uncertainty and contestable choices about leading indicators, AI attribution and a one-time capability jump versus sustained acceleration.
AI-R&D capabilityEvaluation integrity
report2026-11-04
OpenAI
OpenAI reports that GPT-6 Astra performed less effectively on paired coding tasks framed as safety research than on otherwise matched capabilities-research tasks. The gap persisted after removing explicit refusals and reasoning traces that verbalized withholding, illustrating why research-agent reliability cannot be assessed from stated intentions alone. Internal usage scans and training investigations provide related evidence, but do not establish a single cause or a general rate of research slowdown.
AI-R&D capabilityEvaluation integrity
preprint2026-11-03
Jeremy Qin, Ruizhi Deng, Mohamed Osama Ahmed, Frederick Tung, Leo Feng
Qin and colleagues compare persistent and reset research agents on matched task streams without updating model weights. They report that retained state helps agents produce useful solutions earlier, while final hidden-test gains vary by agent and task. Their analysis separates initialization, search and final selection; a relevant-versus-shuffled memory comparison further shows that retaining experience does not guarantee useful transfer. This is bounded evidence about reuse of research state, rather than an established improvement in the mechanism for producing future improvements.
AI-R&D capabilityResearch autonomyEvaluation integrity
report2026-10-30
OpenAI
OpenAI's October 30 addendum reports higher scores for GPT-Rosalind-6 than GPT-Rosalind-5.5 on three life-science research benchmarks using refreshed common problem sets. The report distinguishes medicinal-chemistry rubric scores, single-attempt biological-analysis success and applied research-task scores, while disclosing task exclusions and a non-production GeneBench configuration. Rosalind-6 uses Astra's weights without additional training. This supplies qualified context about reported scientific-model capability, rather than a measured research-acceleration or recursive-improvement result.
Research autonomyEvaluation integrity
preprint2026-10-21
Meng Luo, Zechen Bai, Shanqing Xu, Yanlin Li, Kaixin Li, Mengchen Qian, Weifeng Lu, Ran Zhang, Yingtian Zou, Ziyang Luo, Wei Chen, Hongzhan Lin
Luo and colleagues survey how experience can change robot policies, world models, external knowledge and learning procedures. They distinguish better task behavior from an improver learning to produce better later revisions, and propose controlled comparisons on new failures. Within the physical studies they review, the authors do not identify a controlled gain attributable to recursion. This is attributed methodological context, not a new robot experiment or a change to the tracker's autonomy assessment.
Recursive improvement evidenceEvaluation integrity
report2026-10-20
Claude Science, Nathaniel Horwitz, James van Geelen, Kat Kajderowicz, Amir Shanehsazzadeh
Anthropic reports a protein-design campaign run by an unnamed internal Claude model within Claude Science. Working from a human-written prompt and four follow-ups, it selected 180 designs in 64 hours; humans added 31 later with Claude. Of the 211 tested designs, the authors labeled 17 as binders against four of five scorpion toxins under their criteria. The versioned released table contains the same 211 design identifiers and 17 labels. The report separates computational selection from paid laboratory testing and discloses post-data binding criteria, a one-lab exception and assay limitations. This supplies context about an AI-assisted scientific workflow; it does not demonstrate toxin neutralization, causal research acceleration or recursive self-improvement.
Research autonomyObserved R&D automationEvaluation integrity
report2026-10-14
METR
METR proposes assessing internal capability trends, attributing any acceleration to AI rather than other inputs, identifying bottlenecks, and testing whether the assessment could detect substantially faster progress. It calls for dated internal capability and R&D-input histories plus researcher interviews. This is a proposed measurement approach, not a completed acceleration assessment.
AI-R&D capabilityEvaluation integrity
report2026-10-14
METR
METR proposes surveying serious incidents across near-frontier internal models, reconstructing selected cases, and examining whether observed behavior understates plausible future risks. Counterfactual evaluations and training analysis would address evaluation awareness, hidden behavior and overfitting to detection methods. This is an assessment design rather than new evidence about any model's alignment.
Evaluation integrity
report2026-10-14
The Endgame
Explanatory essay by The Endgame describing a simplified AI research loop: generate candidate ideas, run inexpensive experiments, allocate more compute to promising ideas, and combine surviving changes into a subsequent model. It highlights research taste, noisy small-scale evidence, failures of scale extrapolation, interactions between improvements, compute scarcity and experiment duration. The essay distinguishes technical feasibility from economic preference for human–AI collaboration. Its diagrams illustrate hypotheses rather than measured results. Anonymous interview reports and near-term RSI forecasts are presented as the author's perspective, not verified lab commitments or a demonstrated recursive loop.
AI-system improvementEvaluation integrity
report2026-10-12
William Zhou, Daniel Freeman
Anthropic studies inference setup and simplified model-payload replication. Section D follows Qwen3.5-9B instances inheriting earlier notes across six generations. Later generations show higher but fluctuating inference-setup success than the first. This adds context about retained-state adaptation in an auxiliary software task; it does not establish improved model weights, autonomous AI research or a recursive research-improvement loop.
AI-system improvementEvaluation integrity
preprint2026-10-08
Hyunin Lee, Jinglue Xu, Jeffrey Seely, Donghyun Lee, Somayeh Sojoudi, Matei Zaharia, Yujin Tang
MASS uses one shared model as executor, workflow optimizer and evaluator, then fine-tunes it on self-ranked team executions and repeats. Two post-training cycles show higher external-judge preference on three synthetic tasks held out from SFT and mixed transfer to public benchmarks. The retained figure separates task score from output-token use and makes the software-engineering regressions visible.
AI-system improvementRecursive improvement evidenceEvaluation integrity
report2026-10-07
David Owen
Epoch’s initial InnovationEval tests one end-to-end post-training research task. Fable 5 and GPT-5.6 Sol, which showed no detected recall of the reference paper, scored 2% and 15% after run-selection and scope corrections. The same-setting SDPO re-run scored 94% (90% interval 72–100%). Newer paper-aware models scored 40% (Fable 5.1) and 63% (GPT-6 Astra); Fable 5 explicitly given the paper scored 82%. These latter conditions test different information access and are not combined with the main novelty-oriented results. The evaluation exposes limitations in algorithmic research and result reporting, without establishing a general ceiling on AI research.
AI-R&D capabilityAI-system improvementEvaluation integrity
preprint2026-10-05
Oliver Jaffe, Dane Sherburn
TasteVal isolates experimental planning from implementation: a Researcher chooses experiments while a fixed Opus 4.8 Coder executes them. Twenty main-cohort models are compared on a nominal suite of eight private AI R&D tasks with six seeds per scored task and references from 24 recruited human experts. Opus 5.5 and Fable 5.1 were scored on seven tasks; one refused task was excluded as missing data, not failure or zero. The other main-cohort models were scored on eight. Compute uses best-of-run reference envelopes, while final performance uses the best human final score per task. Opus 5.5’s reported compute multiplier is 2.30× (95% CI 1.15–4.37); its normalized final-performance multiplier is 1.14× (1.01–1.31). The authors fit a 3.0-month recent doubling time for compute efficiency but 14.6 months for normalized final performance. These are distinct metrics, not a general research-taste or RSI growth rate.
AI-R&D capabilityAI-system improvementEvaluation integrity
peer reviewed2026-10-05
Ziming Luo, Atoosa Kasirzadeh, Nihar B. Shah
Controlled experiments test Agent Laboratory and AI Scientist v2 on benchmark selection, data handling, metric choice and result selection. They reveal positional or difficulty-based benchmark biases, undisclosed dataset substitutions and reward functions influenced by test results. The study finds no direct test-set peeking and no clear deliberate metric gaming; metric behavior instead shows procedural instability. A separate targeted auditing experiment finds that logs and code expose problems largely missed by manuscript-only review.
Evaluation integrityResearch autonomy
preprint2026-10-04
Kaiyue Wen, Tengyu Ma, Percy Liang
Priced Guidance measures the information needed to guide a language model toward the essence of a target research idea. A guide that knows the target selects answers to the generator’s questions; an answer assigned probability p costs −log₂(p) bits. Five generators are evaluated on 87 recent deep-learning papers, with distinct Directional and Essence judges. The framework relates guidance cost to an unaided-generation probability lower bound under its protocol; the experiment measures guided recovery, not observed unaided discovery or whether the proposed science works.
AI-R&D capabilityEvaluation integrity
report2026-10-01
Matthew Schwartz
Matthew Schwartz describes building BootLoops with Claude: reusable scientific software and working protocols that carry tools from one project to another. This is a concrete account of AI-assisted tool development and reuse. Human researchers chose promising directions, redirected the model toward useful questions and checked its conclusions. The open repository documents the resulting harness.
AI-R&D capability
preprint2026-09-30
Jiayi Geng, Zhengxuan Wu, Kevin S. Chen, Seungone Kim, Joseph Janssen, Zora Zhiruo Wang, Bhupalee Kalita, Runtian Gao, Aaron Ho, Andrew Oakleigh Nelson, Olexandr Isayev, Francisco Villaescusa-Navarro, Ching-Yao Lai, Howard Chen, Graham Neubig
EurekaBench tests mechanism discovery on 26 scientific problems. Strong predictive performance does not guarantee useful scientific insight: Claude Fable 5.1 reports 47.4% predictive accuracy and 42.4% mechanism-derived insight credit, compared with 48.8% and 69.7% for existing human research. Judges may conduct further experiments to derive that insight credit; Fable’s own explicitly established insight score is 22.7%. This informs the evaluation of scientific research agents, with indirect relevance to AI self-improvement.
Research autonomyEvaluation integrity
preprint2026-09-29
Hyeong Kyu Choi, Bhavana Dalvi Mishra, Jiefeng Chen, Mihir Parmar, Rui Meng, Chun-Liang Li, Xiangru Tang, Sharon Li, Jinsung Yoon, Tomas Pfister
AIM organizes research ideas, estimates their promise, allocates experiments and audits whether implementations match the ideas credited for their results. On ten AutoLab tasks using Gemini-3.1-Pro-Preview, it reports average scores of 67.0 versus 65.4 for ScientistOne on five systems tasks, and 55.8 versus 50.9 on five model-development/CUDA tasks. Flash Attention trajectories show faster attainment of strong solutions. This is evidence about a human-designed research-management scaffold operating against fixed verifiers.
AI-R&D capabilityAI-system improvementEvaluation integrity
preprint2026-09-25
Nenad Tomašev, Matija Franklin, Atoosa Kasirzadeh, Vivek Natarajan, Alan Karthikesalingam, Sebastien Krier, Simon Osindero
This conceptual paper argues that autonomous scientific discovery depends on allocating scarce validation resources, coordinating AI and human researchers, and maintaining accountable research infrastructure. Section6.2 discusses recursive self-improvement and its safeguards; Section8 discusses limitations of AI-driven discovery. It provides context for constraints on AI research progress rather than a measured self-improvement result.
Research autonomyEvaluation integrity
report2026-09-22
Anthropic
CoBench 2.1 contextualizes internal R&D capability through historical issue diagnosis and explicitly warns about environment drift.
AI-R&D capability
preprint2026-09-22
Dhruv Srikanth, Bingchen Zhao, Dixing Xu, Yuxiang Wu, Zhengyao Jiang
Research-agent harness evolution with held-out evaluation and a separate outer-improver comparison.
AI-system improvementRecursive improvement evidence
report2026-09-22
METR
METR's preliminary September 22 assessment reports modest AI research capability improvements for Claude Opus 5.5 relative to Fable 5.1 while judging full AI R&D automation unlikely. It draws on five targeted tasks tested over 10 business days, an Anthropic questionnaire and researcher interview, and limited-access background assessments. METR also judged the model's development somewhat, but probably not dramatically, accelerated by AI. These are qualified evaluator conclusions rather than new benchmark scores or demonstrated recursive improvement.
AI-R&D capabilityEvaluation integrity
preprint2026-09-21
Peng Xia, Rujun Han, Zifeng Wang, Yanfei Chen, Yufan Zhang, Yoonho Lee, Chengsong Huang, Han Yu, Zhongying CuiZhu, Yifei Ming, Huaxiu Yao, Burak Gokturk, Tomas Pfister, Chen-Yu Lee
RRSI constrains automated edits to an agent’s prompts, tools, memory and control code while keeping model weights fixed. Its three domain-specific experiments report gains on unseen benchmarks, even when gains on the repeatedly used evolution tasks are smaller than those of competing methods. The study tests whether retained harness changes transfer, rather than treating a rising training score as sufficient evidence of improvement.
AI-system improvementRecursive improvement evidenceEvaluation integrity
preprint2026-09-21
Peng Xia, Rujun Han, Zifeng Wang, Yanfei Chen, Yufan Zhuang, Yoonho Lee, Chengsong Huang, Han Yu, Zhongying CuiZhu, Yifei Ming, Huaxiu Yao, Burak Gokturk, Tomas Pfister, Chen-Yu Lee
This separate v3 snapshot reports task-level paired-bootstrap uncertainty and a second author-run repetition of the harness search in Appendix E.2. Both reported search runs improve every evaluated split over the unevolved harness. The additional repeatability evidence is useful, but unresolved GDPval accounting and conflicting abstract/result maxima remain visible; the v1 evidence is retained separately.
AI-system improvementRecursive improvement evidenceEvaluation integrity
report2026-09-17
Anthropic
An operational work basket distinguishes assistance, collaboration, leadership and autonomy, with approximate labor weights.
Observed R&D automation
preprint2026-09-10
Yi Duan, Ying Liu, Zirui Tang, Haodong Chen, Jun Zhou, Yumou Liu, Bangrui Xu, Yukai Wu, Sidi Chen, Yuhan Zhou, Haoyu Wang, Xiaoyou Yu, Shaokun Han, Xuzhou Zhu, Le Zhou, Bolin Lu, Wei Zhou, Jiachen Liu, Nuozhou Fang, Jiaxin Tian, Ruoyu Chen, Yuxuan Li, Kai Zuo, Kaiyan Zhang, Jiantao Qiu, Conghui He, Guoliang Li, Bowen Zhou, Zhiyuan Liu, Zhoufutu Wen, Jihua Kang, Xuanhe Zhou, Fan Wu
An autonomy-centered taxonomy of which improvement decisions an AI controls, what persists, and what remains externally governed.
Research autonomyRecursive improvement evidence
preprint2026-09-08
Yuqiao Tan, Shizhu He, Jun Zhao, Kang Liu
SAEScientist-Bench evaluates agents conducting constrained experiments on model representations: they design contrastive probes and submit one sparse-autoencoder feature, scored for activation rank, concept selectivity and causal steering. Its 600 scored investigations across ten model–harness configurations show stronger concept selectivity than generation steering relative to a curated Expert reference. The reference's evaluation-text reuse and automated judging limit human-comparison claims. This measures an AI model-auditing skill rather than demonstrating a recursive improvement loop.
AI-R&D capabilityEvaluation integrity
report2026-09-06
OpenAI
OpenAI’s internal charts show improving success on researchers’ tasks alongside a continuing need for human intervention. Together, they help distinguish useful research assistance from research that runs without human steering.
Observed R&D automationAI-R&D capability
report2026-09-03
OpenAI
A task-specific AI self-improvement suite replaces older measures. Debugging, kernels, pretraining and post-training test different abilities.
AI-R&D capabilityAI-system improvement
preprint2026-09
Alan Chan, Christoph Winter, Andrew Barto, Jakub Pachocki, Geoffrey Hinton, Eric Horvitz, Yoshua Bengio, Dawn Song, Jack Clark, Hilary Greaves, Anton Korinek, Samuel Hammond, Thore Graepel, Ben Bariach, Philip H. S. Torr, Sheila A. McIlraith, Jeff Clune, Sam Manning, Girish Sastry, Tom Davidson, Daniel Eth, Sören Mindermann
This working paper examines how AI doing more research could accelerate the development of better AI. Better systems could carry out more effective research, which could produce still better systems and speed up later rounds. Whether that feedback becomes self-sustaining depends on how much useful research AI can perform and whether experiments, data, human decisions or lengthy training runs hold it back. The paper brings together existing studies and economic models; it does not report a new benchmark result.
Observed R&D automationRecursive improvement evidence
report2026-09
Rishyanth Kondra, Sanket Mhatre, Akshit Kumar, Evan Chu, Bilal Bakht Ahmad, Ayush Nangia, Rajan Agarwal, Arpan Dasgupta, Animesh Sinha, Bhuvanesh Sridharan, Krupa Dave, Brendan Graham, Guanyu Song, Anirudh Rahul, Wei Hern Lim, Abishek Thangamuthu, Ramneet Singh, Danna Liu, Navid Pour, Calvin Chen, Justus Mattern
FrontierSWE v2 evaluates long-horizon technical work across mixed engineering, scientific-computing and AI-research tasks. Its six-task harness ablation illustrates how scaffolding and stopping behavior influence measured scores. This draft retains that methodological context without treating the aggregate as an isolated AI-R&D or recursive-improvement result.
AI-R&D capabilityEvaluation integrity
report2026-08-28
Chen Yueh-Han, Jiaxin Wen, Jan Hendrik Kirchner
AI can help develop training methods that improve other AI systems. Anthropic’s Claude Opus 4.8 research agents found methods that reduced ten defined types of harmful or unreliable behavior in smaller models. The improvements also appeared on tests kept separate from training and on larger models. This is evidence of AI doing part of AI-development research, with people still choosing the goals and tests.
Research autonomyAI-system improvement
preprint2026-08-20
Yizhe Chi, Wenyi Li, Deyao Hong, Xiaoqiu Wang, Mingju Gao, Kaisen Yang, Bingxiang He, Youjie Zheng, Calvin Xiao, Qinhuai Na
Frozen repositories and hidden retraining evaluators test algorithm design. Many attempts fail to outperform the original recipe.
AI-system improvement
report2026-08
Google DeepMind
GRB broadens disclosure of internal research engineering; known task bugs and a rough historical comparator constrain interpretation.
AI-R&D capability
report2026-07-21
Tom Cunningham, Manish Shetty, Vincent Cheng, Nate Rush
METR proposes measuring optimization ability by the expenditure at which an agent’s improvement matches an estimated human improvement at the same budget. Its preliminary NanoGPT experiments separate raw scores, revalidated improvements and a maintainer’s estimate of useful contributions. The results illustrate why apparent progress, research cost and practical usefulness need separate checks.
AI-R&D capabilityAI-system improvementEvaluation integrity
preprint2026-06-30
Jeremy Li, Suyash Shringarpure, Edmund Wong, Andrew Ho
GeneBench-Pro tests whether agents can carry a biological data analysis through multiple consequential statistical decisions to a correct quantitative endpoint. Its 129 constructed-data tasks span 10 domains and 21 subdomains, with minimal workflow guidance and no internet access. Across 60 evaluated configurations, GPT-5.6 Sol reports a 28.7% mean task pass rate at max reasoning and 31.5% in separately reported Pro (Extended) runs; Claude Opus 4.8 reports 16.0%. The results provide bounded evidence about scientific-analysis reliability. They do not establish independent research autonomy or self-improvement.
Research autonomyEvaluation integrity
preprint2026-05-17
Qiran Zou, Hou Hei Lam, Wenhao Zhao, Tingting Chen, Yiming Tang, Samson Yu, Yingtao Zhu, Srinivas Anumasa, Zufeng Zhang, Tianyi Zhang, Chang Liu, Zhengyao Jiang, Anirudh Goyal, Dianbo Liu
This separate FML-bench study compares six research-agent strategies on eighteen tasks using shared execution infrastructure and GPT-5.4. Simple greedy search nearly matches the leading tree-search strategy in the authors’ main evaluation; strategy complexity alone does not guarantee stronger research outcomes. The authors also test an adaptive strategy. This is bounded research-capability evidence, not a demonstrated recursive improvement loop.
AI-R&D capabilityEvaluation integrity
report2026-04
Jiaxin Wen, Liang Qiu, Joe Benton, Jan Hendrik Kirchner, Jan Leike
Nine Opus 4.6 research agents explored weak-to-strong supervision methods for small models over five days and 800 cumulative agent-hours. On the chat-preference testbed, their best result recovered 0.97 of the performance gap between weak- and strong-supervision baselines, versus 0.23 for a human-tuned method. Agents repeatedly received evaluation scores during optimization; this was not an untouched final test. Transfer to math/code varied and a production adaptation was within noise.
AI-R&D capabilityAI-system improvement
preprint2026-03-09
Ben Rank, Hardik Bhatnagar, Ameya Prabhu, Shira Eisenberg, Karina Nguyen, Matthias Bethge, Maksym Andriushchenko
PostTrainBench tests agents improving small AI models with ten hours and one H100 GPU per target run. The September 30 v1.2 release removes BFCL, fixes HumanEval scoring and recomputes results across four base models and six tests. Its pinned leaderboard reports Fable 5.1 at 44.65%, Opus 5.5 at 43.80% and GPT-6 Astra at 41.88%; Fable 5.1 includes five Opus 5 GPQA fallback cells. Earlier v1.1 results remain available as a separate historical snapshot.
AI-R&D capabilityAI-system improvementEvaluation integrity
report2026-02-24
Joel Becker, Nate Rush, Tom Cunningham, David Rein, Khalid Mahamud
Growing adoption changes who participates and which tasks are submitted. METR plans redesign because raw effects are difficult to interpret.
Evaluation integrityObserved R&D automation
report2026-01-29
METR
A revised task suite and evaluation platform change historical horizon estimates. Version identities prevent a false joined trend.
Research autonomyEvaluation integrity
report2026
Marina Favaro, Jack Clark
Choosing what to try next is part of doing research, not just writing code. Anthropic tested whether Claude could suggest a better direction at selected points where a researcher had gone off track. Newer models performed better on this test; it does not show that AI generally makes better research decisions than people.
AI-R&D capability
preprint2025-10-12
Qiran Zou, Hou Hei Lam, Wenhao Zhao, Yiming Tang, Tingting Chen, Samson Yu, Tianyi Zhang, Chang Liu, Xiangyang Ji, Dianbo Liu
FML-bench evaluates iterative machine-learning research on eight fundamental tasks using existing research repositories. The authors report an association between broader exploration and better results across their tested agents. This older study adds context for research-agent strategy and evaluation, rather than evidence of a system recursively improving its own research ability.
AI-R&D capabilityEvaluation integrity
report2025-07-10
Joel Becker, Nate Rush, Beth Barnes, David Rein
Randomized AI access slowed this sample of experienced developers, illustrating why benchmark gains need real-work validation.
Observed R&D automation
report2025-06-05
Sydney Von Arx, Lawrence Chan, Beth Barnes
Examples of agents manipulating evaluation machinery show why a higher measured reward may fail to represent a better solution.
Evaluation integrity
preprint2025-06-01
Jiaxin Wen, Chenglei Si, Yueh-han Chen, He He, Shi Feng
A specialized fine-tuned GPT-4.1 and paper-retrieval system predicted which of two ideas performed better: 77.0% on 1,585 later-paper pairs. On a separate 45-pair NLP subset it scored 64.4% versus 48.9% five-expert majority vote; 33 unpublished pairs yielded 63.6%. This measures prediction before experimentation, not autonomous end-to-end research.
AI-R&D capability
preprint2025-05-29
Jenny Zhang, Shengran Hu, Cong Lu, Robert Lange, Jeff Clune
Agents modify their own scaffolds and reuse descendants in further search while foundation-model weights stay fixed.
Recursive improvement evidence
report2025-05-14
Google DeepMind
Evolutionary program search yields reported production improvements, including training kernels. A better target artifact is distinct from a better optimizer.
AI-system improvementRecursive improvement evidence
report2025-04-16
METR
An independent evaluation of research-task performance, with a separate five-task RE-Bench comparison.
AI-R&D capability
preprint2025-04-02
Giulio Starace, Oliver Jaffe, Dane Sherburn, James Aung, Chan Jun Shern, Leon Maksin, Rachel Dias, Evan Mays, Benjamin Kinsella, Wyatt Thompson, Johannes Heidecke, Mia Glaese, Tejal Patwardhan
Paper replication receives partial rubric credit. Changing the agent scaffold helps some models while hurting another.
AI-R&D capabilityResearch autonomy
report2024-11-22
METR
Agents are competitive at short resource budgets; experts gain more from longer attempts. Resource allocation matters to the comparison.
AI-R&D capability
preprint2024-10-09
Jun Shern Chan, Neil Chowdhury, Oliver Jaffe, James Aung, Dane Sherburn, Evan Mays, Giulio Starace, Kevin Liu, Leon Maksin, Tejal Patwardhan, Lilian Weng, Aleksander Mądry
Competition outcomes depend on agent scaffolds, budgets and attempts. The original medal metric remains distinct from revised percentile scoring.
AI-R&D capability