preprint · 2026-10-08

Recursive Self-Improvement through Multi-Agent Self-Supervision

MASS authors (UC Berkeley and Sakana AI) · Hyunin Lee, Jinglue Xu, Jeffrey Seely +4 more

On this page

Why it matters here

MASS uses one shared model as executor, workflow optimizer and evaluator, then fine-tunes it on self-ranked team executions and repeats. Two post-training cycles show higher external-judge preference on three synthetic tasks held out from SFT and mixed transfer to public benchmarks. The retained figure separates task score from output-token use and makes the software-engineering regressions visible.

Read original source ↗

AI-system improvementRecursive improvement evidenceEvaluation integrity

What the charts show

MASS: uneven public-benchmark transfer after two self-training rounds

The printed round-two score-per-output-token ratios are 1.60 for MLR-Bench, 1.35 for DSBench, 1.20 for ScienceAgentBench and 1.28 for AstaBench E2E-Hard, versus 0.96 for Terminal-Bench 2.0 and 0.94 for SWE-bench Verified. This supports bounded and uneven transfer, not a general research-speedup ratio.

Qwen3.6-27B-FP8 base and two MASS generations use qwen-code 0.20.0 with common tools and bare benchmark instructions. Axes pair mean output tokens per task with benchmark-specific score. Blue/orange/green denote base/round one/round two; dashed lines mark the base score-per-token ratio.

View original chart and methods ↗
Recursive Self-Improvement through Multi-Agent Self-Supervision · Paper Figure 2 and Appendix C.5 Table 7; project Figure 3

How to interpret this chart

Author-reported preprint; source extraction and original Figure 2 inspection, not independent evaluation or experimental replication.
Only two completed post-training cycles on a human-designed synthetic suite: nine designated training tasks, eight used for SFT, and three synthetic test tasks. GPT-5.5 generated task descriptions before the experiment.
Synthetic test tasks are excluded from SFT, validation and checkpoint selection, but workflow search is also evaluated on them. Six external model judgments per workspace pair are repeated verdicts on the same executions, not six independent research attempts.
Public-benchmark transfer is uneven. Score per output token excludes training, workflow search and evaluation costs and is not end-to-end research speedup. Error bars are sample standard deviations across trial means, not confidence intervals.
MLR-Bench and AstaBench aggregate gains accompany more delivered reports. Conditional report-quality means concern different completed sets; they do not establish a quality decline or gain on a common completed subset. MLR-Bench paired task sign tests are not significant even where trial-level tests are.
ScienceAgentBench reference programs succeed on only 80.4% in the local environment. Terminal-Bench and SWE-bench scoring excludes tasks without verifier results for any compared model, with trial-specific matched denominators; agent timeouts are not rerun.
Workflow-transfer evaluation holds executor weights fixed, but the compared workflows were discovered using different executors and evaluators. It does not isolate optimizer improvement.
The paper prose reports ScienceAgentBench 28.7% to 31.8%, whereas Appendix C.5 Table 7 and the explicit trial counts support 28.8% to 31.7%. This proposal uses Table 7 and preserves that source inconsistency.
No evidence here establishes unbounded compounding, autonomous frontier research, a tracker autonomy level, or total compute-matched recursive acceleration.

Published values from the chart

Exact Appendix C.5 Table 7 values and reported trial means/sample standard deviations, not estimates from plot marks. Benchmark means use their distinct scoring rules. MLR missing papers and Asta missing reports receive zero; papers delivered 31/60,36/60,50/60 and reports delivered 60/120,69/120,91/120. These completion counts are not independent task samples.

MASS: uneven public-benchmark transfer after two self-training rounds
Benchmark / metric	Base	Round one	Round two	Tasks × trials
ScienceAgentBench / success (%)	28.8 ± 1.5	30.4 ± 2.0	31.7 ± 2.0	102 × 3
MLR-Bench / Overall (0–10)	1.58 ± 0.38	1.80 ± 0.30	2.40 ± 0.28	12 × 5
AstaBench E2E-Bench-Hard / rubric fraction	0.053 ± 0.002	0.055 ± 0.004	0.067 ± 0.017	40 × 3
DSBench / normalized score	0.473 ± 0.029	0.465 ± 0.011	0.482 ± 0.025	74 × 3
Terminal-Bench 2.0 / pass (%)	37.5 ± 2.2	35.3 ± 2.1	35.2 ± 1.7	89 scheduled × 3; scored 86/88/87
SWE-bench Verified / pass (%)	62.6 ± 1.4	61.8 ± 1.9	61.1 ± 0.6	500 scheduled × 3; scored 499/499/496

What to keep in mind

Author-reported preprint; source extraction and original Figure 2 inspection, not independent evaluation or experimental replication.
Only two completed post-training cycles on a human-designed synthetic suite: nine designated training tasks, eight used for SFT, and three synthetic test tasks. GPT-5.5 generated task descriptions before the experiment.
Synthetic test tasks are excluded from SFT, validation and checkpoint selection, but workflow search is also evaluated on them. Six external model judgments per workspace pair are repeated verdicts on the same executions, not six independent research attempts.
Public-benchmark transfer is uneven. Score per output token excludes training, workflow search and evaluation costs and is not end-to-end research speedup. Error bars are sample standard deviations across trial means, not confidence intervals.
MLR-Bench and AstaBench aggregate gains accompany more delivered reports. Conditional report-quality means concern different completed sets; they do not establish a quality decline or gain on a common completed subset. MLR-Bench paired task sign tests are not significant even where trial-level tests are.
ScienceAgentBench reference programs succeed on only 80.4% in the local environment. Terminal-Bench and SWE-bench scoring excludes tasks without verifier results for any compared model, with trial-specific matched denominators; agent timeouts are not rerun.
Workflow-transfer evaluation holds executor weights fixed, but the compared workflows were discovered using different executors and evaluators. It does not isolate optimizer improvement.
The paper prose reports ScienceAgentBench 28.7% to 31.8%, whereas Appendix C.5 Table 7 and the explicit trial counts support 28.8% to 31.7%. This proposal uses Table 7 and preserves that source inconsistency.
No evidence here establishes unbounded compounding, autonomous frontier research, a tracker autonomy level, or total compute-matched recursive acceleration.

Study details

Bounded evidence of repeated self-supervised model updates and inherited use across roles. This study does not establish isolated optimizer improvement, total-budget-matched acceleration, or sustained unbounded progress.

Methods and study design

Study design and reported evidence
What changed	weights, training_data, scaffold, prompt
What stayed fixed	qwen-code 0.20.0 coding-agent environment; Human-designed synthetic task suite and experimental procedure
Improved system used as optimizer later	yes
Generations attempted / accepted	2 / 2
Held-out transfer	Three synthetic tasks are held out from SFT, validation and checkpoint selection but also used to evaluate workflow search. Public-benchmark results show gains on research-task aggregates and lower Terminal-Bench/SWE-bench scores.
Resource accounting	Score per output token is reported separately from supervised training-target tokens. Selected round-one checkpoint processes 28.82M supervised targets; excluded costs include teacher generation, workflow search, validation and evaluation.
Human contributions	Humans choose the base model, task suite, procedure, tools, budgets, LoRA recipe and evaluation rules; GPT-5.5 generates synthetic task descriptions before the experiment. The shared current model supplies MASS workflow feedback and trajectory ranking, while external model verdicts are withheld from the main loop.
Author claims	The authors describe homogeneous recursive self-improvement and transfer from task-solving supervision to workflow optimization and evaluation.