operational metric

Experienced developer productivity

Randomized access to AI tools on real repository issues.

Observed R&D automation

What this measure tells us

Operational counterpoint to benchmark performance.

Experienced developers in familiar open-source repositories; not representative of all work.
Later study reports selection bias, so no unqualified trend.

Benchmark version

Metric

Results to compare

Protocol

2026-02-24

Selection effects prevent a reliable current effect estimate.

Interpretability assessment (qualitative): A task-specific measurement; not a percentage of RSI achieved.

AI-assisted task-time change (percent): Negative means faster completion. Not a reliable population effect because of selection bias.

Published results · Late-2025 study update
Key	System / organization	Metric	Reported result	Date	Protocol	Verification	Evidence
—	METR10 returning and 47 newly recruited developers	Interpretability assessment	METR reports that selection effects make the later experiment unreliable for estimating current productivity gains.	2026-02-24	productivity-later-p	independent evaluation	We are Changing our Developer Productivity Experiment Design ↗
—	METRReturning original-study participants	AI-assisted task-time change	-18%Confidence interval: -38% to +9% · Source-reported confidence interval; level not specified in summary	2026-02-24	productivity-returning-p	independent evaluation	We are Changing our Developer Productivity Experiment Design ↗
—	METRNewly recruited developers	AI-assisted task-time change	-4%Confidence interval: -15% to +9% · Source-reported confidence interval; level not specified in summary	2026-02-24	productivity-new-p	independent evaluation	We are Changing our Developer Productivity Experiment Design ↗

2025-07-10 · 246 tasks

16 developers; issue-level randomization.

Change in completion time (percent): A task-specific measurement; not a percentage of RSI achieved.

Published results · Early-2025 randomized trial
Key	System / organization	Metric	Reported result	Date	Protocol	Verification	Evidence
—	METR16 experienced open-source developers	Change in completion time	19% longer	2025-07-10	productivity-p	independent evaluation	Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity ↗

No reference points are indexed for this measure.