Progress

benchmark

PostTrainBench

Tests how well AI agents improve small AI models, with ten hours and one H100 GPU per training run.

On this page
AI-R&D capabilityAI-system improvementEvaluation integrity

What this measure tells us

Tests bounded AI post-training research and experimental iteration.

  • Version 1.2 combines four base models and six target tests; historical v1.1 uses seven. Scores across these versions are not directly comparable.
  • Google’s Gemini 4 Argon report uses a separate v1.1/OpenCode setup; keep those results distinct from the benchmark-author leaderboard.
  • Each run optimizes one assigned model for one target test. A high aggregate does not establish broad transfer or improvement of the research agent itself.
  • Fable 5.1 includes five GPQA run cells from Opus 5; the result is a mixed-system aggregate.

Results by version

PostTrainBench v1.1 / Google OpenCode evaluation, October 2026

2026-10

Google reports self-computed results for all four evaluated systems, using OpenCode, ten hours and one NVIDIA H100. Separate disclosure snapshot from the benchmark-author leaderboard. The September 30 announcement links this PDF, whose results heading says October 2026. Exact evaluation dates, seeds, weights and audit outcomes are not disclosed here.

Metric definitions

Weighted downstream benchmark score (percent): Reported downstream performance after bounded post-training, not normalized improvement, an RSI percentage or proof of an integrated recursive-improvement system.

Published results · PostTrainBench v1.1 / Google OpenCode evaluation, October 2026
KeySystem / organizationMetricReported resultDateProtocolVerificationEvidence
—Gemini 4 Argon / Google OpenCode evaluationWeighted downstream benchmark score45.3%2026-10ptb-google-argon-202610-protocollab reported
—GPT-6 Astra / Google OpenCode evaluationWeighted downstream benchmark score44.3%2026-10ptb-google-argon-202610-protocollab reported
—Claude Fable 5.1 / Google OpenCode evaluationWeighted downstream benchmark score40.2%2026-10ptb-google-argon-202610-protocollab reported
—Claude Opus 5.5 / Google OpenCode evaluationWeighted downstream benchmark score49.3%2026-10ptb-google-argon-202610-protocollab reported

PostTrainBench v1.2 · September 30 release

2026-09-30 · 24 tasks

Version 1.2 removes BFCL, reweights six tests, fixes HumanEval scoring and changes contamination judging to a two-of-three majority. Final-model evaluations average five seeds for AIME, three for GSM8K/GPQA/HumanEval and one for ArenaHard/HealthBench. These evaluation repetitions are separate from aggregate score seeds. Results combine four base models and six targets (24 configurations per score seed). The authors recomputed prior results; a score change from v1.1 is not itself model improvement. Snapshot pinned October 1.

Metric definitions

Weighted downstream benchmark score (percent): Downstream target performance after bounded post-training, including source-defined fallback scores. This is not normalized gain, a general model-quality score or a percentage of RSI achieved.

Published results · PostTrainBench v1.2 · September 30 release
KeySystem / organizationMetricReported resultDateProtocolVerificationEvidence
—Claude Fable 5.1 + Opus 5 GPQA fallback / Claude Code, MaxWeighted downstream benchmark score44.65%Standard deviation across score seeds: ± 2.72 percentage points across 2 score seeds2026-09-30ptb-v12-fable-5-1-protocolbenchmark author reported
—Claude Opus 5.5 / Claude Code, MaxWeighted downstream benchmark score43.80%Standard deviation across score seeds: ± 3.00 percentage points across 2 score seeds2026-09-30ptb-v12-opus-5-5-max-protocolbenchmark author reported
—GPT-6 Astra / Codex CLI, MaxWeighted downstream benchmark score41.88%Standard deviation across score seeds: ± 0.76 percentage points across 2 score seeds2026-09-30ptb-v12-gpt-6-astra-protocolbenchmark author reported
—Locus (Intology, powered by Opus 5)Weighted downstream benchmark score41.66%Standard deviation across score seeds: ± 0.82 percentage points across 3 score seeds2026-09-30ptb-v12-locus-protocolbenchmark author reported
—Fable 5 + Opus 4.8 Max GPQA fallbackWeighted downstream benchmark score37.27%Standard deviation across score seeds: ± 1.65 percentage points across 2 score seeds2026-09-30ptb-v12-fable-5-protocolbenchmark author reported
—GLM 5.3 Flash / Claude Code, MaxWeighted downstream benchmark score36.79%Standard deviation across score seeds: ± 0.11 percentage points across 2 score seeds2026-09-30ptb-v12-glm-5-3-flash-protocolbenchmark author reported
—Claude Opus 5 / Claude CodeWeighted downstream benchmark score35.66%Standard deviation across score seeds: ± 2.62 percentage points across 2 score seeds2026-09-30ptb-v12-opus-5-protocolbenchmark author reported
—GLM 5.3 / Claude Code, MaxWeighted downstream benchmark score34.52%Standard deviation across score seeds: ± 3.39 percentage points across 2 score seeds2026-09-30ptb-v12-glm-5-3-protocolbenchmark author reported
—Kimi K3, 1M context / Claude CodeWeighted downstream benchmark score31.48%Standard deviation across score seeds: ± 0.63 percentage points across 3 score seeds2026-09-30ptb-v12-kimi-k3-protocolbenchmark author reported
—GPT-5.6 Sol, Max reasoning / Codex CLIWeighted downstream benchmark score31.01%Standard deviation across score seeds: ± 0.20 percentage points across 2 score seeds2026-09-30ptb-v12-gpt-5-6-sol-protocolbenchmark author reported
—Claude Opus 4.8, Max reasoning / Claude CodeWeighted downstream benchmark score31.00%Standard deviation across score seeds: ± 3.00 percentage points across 2 score seeds2026-09-30ptb-v12-opus-4-8-max-protocolbenchmark author reported
—Claude Opus 4.8, High reasoning / Claude CodeWeighted downstream benchmark score30.66%Standard deviation across score seeds: ± 3.04 percentage points across 2 score seeds2026-09-30ptb-v12-opus-4-8-protocolbenchmark author reported
—GLM 5.2, Max reasoning / Claude CodeWeighted downstream benchmark score30.40%Standard deviation across score seeds: ± 1.46 percentage points across 3 score seeds2026-09-30ptb-v12-glm-5-2-protocolbenchmark author reported
—Claude Opus 4.7, xHigh reasoning / Claude CodeWeighted downstream benchmark score24.34%Standard deviation across score seeds: ± 1.63 percentage points across 3 score seeds2026-09-30ptb-v12-opus-4-7-protocolbenchmark author reported
—GPT-5.5, xHigh reasoning / Codex CLIWeighted downstream benchmark score23.87%Standard deviation across score seeds: ± 0.48 percentage points across 2 score seeds2026-09-30ptb-v12-gpt-5-5-xhigh-protocolbenchmark author reported
—Grok 4.5, High reasoning / Cursor CLIWeighted downstream benchmark score22.62%Standard deviation across score seeds: ± 0.94 percentage points across 2 score seeds2026-09-30ptb-v12-grok-4-5-high-protocolbenchmark author reported
—Gemini 3.1 Pro / OpenCodeWeighted downstream benchmark score18.11%Standard deviation across score seeds: ± 1.68 percentage points across 3 score seeds2026-09-30ptb-v12-gemini-3-1-pro-protocolbenchmark author reported
—GPT-5.4, High reasoning / Codex CLIWeighted downstream benchmark score17.76%Standard deviation across score seeds: ± 1.80 percentage points across 3 score seeds2026-09-30ptb-v12-gpt-5-4-high-protocolbenchmark author reported

PostTrainBench v1.1 · September 2026 results

2026-07-28 · 28 tasks

Each result combines four base models and seven tests (28 target combinations per seed). These include BFCL v3 exec_simple, ArenaHard v2 creative writing, and a selected 245-question HealthBench-Easy set, rather than those entire benchmarks. The v1.1 release kept the original benchmark tasks and final evaluation. Results combine original runs that passed updated checks and reruns. Historical v1.1 results precede the September 30 v1.2 removal of BFCL, HumanEval scoring fix, repeated final evaluations and changed judging. Retained for history; do not compare a v1.1-to-v1.2 score difference as capability growth.

Metric definitions

Weighted downstream benchmark score (percent): Reported target-evaluation performance after a bounded post-training run, including official base-score fallback for flagged/missing cells. Unlike PostTrainBench Lite, this is not normalized reward.

Published results · PostTrainBench v1.1 · September 2026 results
KeySystem / organizationMetricReported resultDateProtocolVerificationEvidence
—Fable 5 + Opus 4.8 Max GPQA fallbackWeighted downstream benchmark score41.79%Standard deviation across score seeds: ± 1.7% across 2 seeds2026-09-18ptb-v11-fable-5-protocolbenchmark author reported
—GLM 5.2, Max reasoning / Claude CodeWeighted downstream benchmark score31.70%Standard deviation across score seeds: ± 2.1% across 3 seeds2026-09-18ptb-v11-glm-5-2-protocolbenchmark author reported
—GPT-5.4, High reasoning / Codex CLIWeighted downstream benchmark score19.00%Standard deviation across score seeds: ± 3.0% across 3 seeds2026-09-18ptb-v11-gpt-5-4-high-protocolbenchmark author reported
—GPT-5.5, xHigh reasoning / Codex CLIWeighted downstream benchmark score27.23%Standard deviation across score seeds: ± 0.6% across 2 seeds2026-09-18ptb-v11-gpt-5-5-xhigh-protocolbenchmark author reported
—GPT-5.6 Sol, Max reasoning / Codex CLIWeighted downstream benchmark score36.23%Standard deviation across score seeds: ± 0.2% across 2 seeds2026-09-18ptb-v11-gpt-5-6-sol-protocolbenchmark author reported
—Gemini 3.1 Pro / OpenCodeWeighted downstream benchmark score21.99%Standard deviation across score seeds: ± 1.8% across 3 seeds2026-09-18ptb-v11-gemini-3-1-pro-protocolbenchmark author reported
—Grok 4.5, High reasoning / Cursor CLIWeighted downstream benchmark score23.45%Standard deviation across score seeds: ± 0.1% across 2 seeds2026-09-18ptb-v11-grok-4-5-high-protocolbenchmark author reported
—Kimi K3, 1M context / Claude CodeWeighted downstream benchmark score31.96%Standard deviation across score seeds: ± 0.3% across 3 seeds2026-09-18ptb-v11-kimi-k3-protocolbenchmark author reported
—Claude Opus 4.7, xHigh reasoning / Claude CodeWeighted downstream benchmark score28.56%Standard deviation across score seeds: ± 1.7% across 3 seeds2026-09-18ptb-v11-opus-4-7-protocolbenchmark author reported
—Claude Opus 4.8, High reasoning / Claude CodeWeighted downstream benchmark score33.84%Standard deviation across score seeds: ± 3.6% across 2 seeds2026-09-18ptb-v11-opus-4-8-protocolbenchmark author reported
—Claude Opus 4.8, Max reasoning / Claude CodeWeighted downstream benchmark score32.90%Standard deviation across score seeds: ± 5.8% across 2 seeds2026-09-18ptb-v11-opus-4-8-max-protocolbenchmark author reported
—Claude Opus 5 / Claude CodeWeighted downstream benchmark score35.04%Standard deviation across score seeds: ± 1.4% across 2 seeds2026-09-18ptb-v11-opus-5-protocolbenchmark author reported
—Locus (Intology, powered by Opus 5)Weighted downstream benchmark score45.58%Standard deviation across score seeds: ± 0.8% across 3 seeds2026-09-18ptb-v11-locus-protocolbenchmark author reported

Version lineage

  1. PostTrainBench v1.1 · September 2026 results
  2. PostTrainBench v1.1 / Google OpenCode evaluation, October 2026
  3. PostTrainBench v1.2 · September 30 release

Reference points

Availability

What is publicly available
ResourceStatus
public descriptionyes
public resultsyes
public tasksyes
public codeyes
public evaluation serviceunknown

Official sources