Evidence record
GRB internal research engineering
August 2026 report snapshot · Gemini 3.1 Pro / preliminary GRB runs · 2026-08
On this page
Reported result · numeric
around 16%
Average pass@1 · percent
lab reported · extraction review: agent checked
Sources
Published 2026-08
Evaluation setup
| comparability caveats | No detailed transcript analysis of older runs. |
|---|
Not reported: split, task count, task snapshot, attempts per task, run count, selection rule, aggregate method, token budget, wall clock budget, hardware, training budget, inference budget, monetary cost, tool access, internet access, filtering, scaffold, evaluator version, human intervention, task exclusions, contamination concerns.
Comparability
limited comparison
- Rough older estimate; no automatic delta.
Limitations
- Approximate preliminary estimate, not a confident baseline.
- Approximately 20% of tasks had bugs that could cause false negatives.
- Older result is a preliminary estimate.