Progress

benchmark

Internal Research Debugging Evaluation

Resolve real bugs in internal research experiments.

AI-R&D capability

What this measure tells us

Targets a specific AI-development task.

  • Internal suite; no cross-lab percentage comparison.
  • Not a demonstration of recursive improvement.

Results by version

Astra card snapshot, September 2026

2026-09-03

Disclosure snapshot. The source describes 41 research bugs and six alignment-auditing tasks, but does not explicitly identify the plotted mean’s scored denominator.

Metric definitions

Mean rubric reward (percent): A task-specific measurement; not a percentage of RSI achieved.

Internal Research Debugging Evaluation · Mean rubric reward

limited comparison

Internal task set and rubric are not public; no uncertainty intervals are printed.

Internal Research Debugging Evaluation: Mean rubric rewardCategorical point plot with no connecting line. Astra Appendix Figure 84 reports all five models on the same labeled internal evaluation.406080percent2026-09-03: 50.8% · GPT-5.6 Luna150.82026-09-03: 68.32% · GPT-5.6 Sol268.322026-09-03: 46.62% · GPT-6 Luna346.622026-09-03: 64.20% · GPT-6 Sol464.22026-09-03: 78.05% · GPT-6 Astra578.05
Published results · Astra card snapshot, September 2026
KeySystem / organizationMetricReported resultDateProtocolVerificationEvidence
5GPT-6 AstraMean rubric reward78.05%2026-09-03openai-research-debugging-protocollab reported
4GPT-6 SolMean rubric reward64.20%2026-09-03openai-research-debugging-protocollab reported
2GPT-5.6 SolMean rubric reward68.32%2026-09-03openai-research-debugging-protocollab reported
3GPT-6 LunaMean rubric reward46.62%2026-09-03openai-research-debugging-protocollab reported
1GPT-5.6 LunaMean rubric reward50.8%2026-09-03openai-research-debugging-protocollab reported

Version lineage

Astra card snapshot, September 2026

Reference points

other

Indicative High capability threshold

OpenAI reports 78.05% as below its indicative High capability threshold. A numeric cutoff has not been verified in this record. This is not an RSI threshold.

What this reference means: unknown · Where it applies: established

Qualitative source assessment only; do not infer a numeric cutoff or distance to the threshold.

Availability

What is publicly available
ResourceStatus
public descriptionyes
public resultsyes
public tasksno
public codeunknown
public evaluation serviceunknown

Official sources