InnovationEval

What this measure tells us

Directly tests an important ingredient of AI self-improvement: end-to-end algorithmic research on AI training. The first task targets SDPO-like gains over GRPO while post-training Qwen3-8B.

• One research task based on one human paper, with one main agent attempt per model; this does not establish general research reliability.
• The 90% confidence intervals concern seed/rerun noise in training the submitted method, not variability across independent agent attempts.
• No detected paper recall is not proof of uncontaminated training: GPT-5.6 Sol’s cutoff overlaps the paper’s January 28 publication.
• Human reviewers assessed submissions, logs and transcripts with LLM assistance. Scope corrections were estimated from existing experiments; footnote 15 says this preliminary report did not require fresh re-grading with scope violations ablated.
• The coding task is transductive: training and evaluation use the same 131 problems with different unit-test access. Test splits were visible for selection.
• Coding scope correction uses comparable wall-clock time, although the task prompt specifies a generation-based axis; Epoch acknowledges this judgment in footnote 20.
• This is normalized task performance, not percent of all AI research automated, a direct novelty grade, a matched human research success rate, or evidence of recursive acceleration.