preprint · 2026-05-17
FML-bench: A Controlled Study of AI Research Agent Strategies from the Perspective of Search Dynamics
On this page
Why it matters here
This separate FML-bench study compares six research-agent strategies on eighteen tasks using shared execution infrastructure and GPT-5.4. Simple greedy search nearly matches the leading tree-search strategy in the authors’ main evaluation; strategy complexity alone does not guarantee stronger research outcomes. The authors also test an adaptive strategy. This is bounded research-capability evidence, not a demonstrated recursive improvement loop.
AI-R&D capabilityEvaluation integrity
What to keep in mind
- Author-reported preprint and source extraction; no independent replication.
- Three 100-step rounds per agent/task; a step is a validation-command execution, excluding internal model calls. Only validation feedback reaches agents; the best-validated codebase is tested afterward.
- The main report averages three runs; Appendix L selects the best of three. The ported strategies are not complete original product configurations or compute-matched systems.
- Opportunity-density and process correlations do not establish a universal causal strategy rule. The adaptive strategy is author-designed, not evidence of autonomous optimizer improvement.
- The task population and protocol differ from the 2025 breadth study. No merged trend, numeric benchmark rows or new figure interpretation is added.