MLE-bench | RSI Progress

What this measure tells us

Measures practical training and data-science work.

Revised 2026 tasks and percentile scoring are incompatible with 2024 medal rates.
Public competition history creates contamination risk.

Results by version

Benchmark version

Metric

Results to compare

Protocol

Revised, Astra card snapshot

2026-09-03 · 72 tasks

Replaces tasks and scores percentile rank against a generated reference distribution; up to three leaderboard submissions.

Metric definitions

Mean percentile against reference solutions (percent): Percentile rank against a generated distribution of strong solutions; not the percentage of competitions earning a medal.

Published results · Revised, Astra card snapshot
Key	System / organization	Metric	Reported result	Date	Protocol	Verification	Evidence
—	GPT-6 Astra	Mean percentile against reference solutions	93.80%	2026-09-03	mle-revised-max	lab reported	GPT-6 Astra System Card ↗
—	GPT-5.6 Sol	Mean percentile against reference solutions	87.18%	2026-09-03	mle-revised-max	lab reported	GPT-6 Astra System Card ↗

Original paper v1 (2024)

2024-10-09 · 75 tasks

Mean any-medal fraction per attempt; not pass@k.

Metric definitions

Any medal rate (percent): A task-specific measurement; not a percentage of RSI achieved.

MLE-bench · Any medal rate

direct comparison · Point estimates; uncertainty is shown in the table.

Different seed counts affect precision; no significance inference from rounded means.

Scroll sideways to see the whole chart

Published results · Original paper v1 (2024)
Key	System / organization	Metric	Reported result	Date	Protocol	Verification	Evidence
1	GPT-4o / AIDE	Any medal rate	8.7 ± 0.5%Standard error	2024-10-09	mle-4o-p	benchmark author reported	MLE-bench: Evaluating Machine Learning Agents on Machine Learning Engineering (v1)↗
2	Claude 3.5 Sonnet / AIDE	Any medal rate	7.6 ± 1.8%Standard error	2024-10-09	mle-claude-p	benchmark author reported	MLE-bench: Evaluating Machine Learning Agents on Machine Learning Engineering (v1)↗
3	Llama 3.1 405B / AIDE	Any medal rate	3.0 ± 1.0%Standard error	2024-10-09	mle-llama-p	benchmark author reported	MLE-bench: Evaluating Machine Learning Agents on Machine Learning Engineering (v1)↗
4	o1-preview / AIDE	Any medal rate	16.9 ± 1.1%Standard error	2024-10-09	mle-o1-p	benchmark author reported	MLE-bench: Evaluating Machine Learning Agents on Machine Learning Engineering (v1)↗
—	GPT-4o / MLAB	Any medal rate	0.8 ± 0.5%Standard error	2024-10-09	mle-4o-mlab-p	benchmark author reported	MLE-bench: Evaluating Machine Learning Agents on Machine Learning Engineering (v1)↗
—	GPT-4o / OpenHands	Any medal rate	4.4 ± 1.4%Standard error	2024-10-09	mle-4o-openhands-p	benchmark author reported	MLE-bench: Evaluating Machine Learning Agents on Machine Learning Engineering (v1)↗

Version lineage

Reference points

No reference points are indexed for this measure.

Availability

What is publicly available
Resource	Status
public description	yes
public results	yes
public tasks	yes
public code	yes
public evaluation service	unknown

Official sources

MLE-bench: Evaluating Machine Learning Agents on Machine Learning Engineering (v1)Sections 2–3; Table 2Original source ↗