Add a benchmark

Benchmarks / Coding

ProgramBench v1 Almost Resolved

Share of public ProgramBench v1 tasks where the implementation passes at least 95% of hidden behavioral tests.

62models
–Open Index weight
2026-10-12last updated
ProgramBenchpublisher

ProgramBench v1 Almost Resolved

Higher is better · best effort per model

Chart settings
62 of 62 selected

full table below

#ModelEffortScoreCost / taskHarness
1Opus 5.5max65.0%$68.05mini-SWE-agent
2Fable 5.1max53.5%$58.52mini-SWE-agent
3GPT-6 Astramax50.0%$11.57mini-SWE-agent
4Sonnet 5.5max49.5%$35.86mini-SWE-agent
5GPT-6.1 Solmax46.0%$2.87mini-SWE-agent
6Opus 5max41.5%$60.29mini-SWE-agent
7Gemini 4high41.0%$17.66mini-SWE-agent
8Fable 5max33.0%$75.68mini-SWE-agent
9GPT-6 Solmax32.5%$11.67mini-SWE-agent
10Haiku 5.5max29.5%$6.83mini-SWE-agent
11GPT-5.6 Solmax23.0%$15.29mini-SWE-agent
12Grok 4.7xhigh21.5%$46.49mini-SWE-agent
13Grok 4.6high19.0%$16.76mini-SWE-agent
14Muse Spark 1.3max19.0%$54.29mini-SWE-agent
15Kimi K3max17.5%$70.48mini-SWE-agent
16MiMo V2.6 Flashhigh17.5%$0.471mini-SWE-agent
17DeepSeek V4.1 Flashmax16.5%$0.897mini-SWE-agent
18GLM 5.3max16.5%$21.96mini-SWE-agent
19Opus 4.8max15.5%$31.27mini-SWE-agent
20GPT-5.6 Terramax15.0%$5.38mini-SWE-agent
21Hy4–14.5%$12.81mini-SWE-agent
22Gemini 3.8 Flashhigh14.0%$10.93mini-SWE-agent
23Sonnet 5max13.5%$24.36mini-SWE-agent
24DeepSeek V4 Pro 0813max13.5%$0.432mini-SWE-agent
25MiMo V2.6 Prohigh12.5%$0.727mini-SWE-agent
26Qwen 3.8 Maxmax10.5%$12.36mini-SWE-agent
27Opus 4.7max9.5%–mini-SWE-agent
28Gemini 3.7 Flashhigh9.5%$7.13mini-SWE-agent
29GPT-5.5xhigh9.5%$6.95mini-SWE-agent
30GLM 5.2max9.5%$12.58mini-SWE-agent
31GPT-5.6 Lunamax9.0%$0.943mini-SWE-agent
32DeepSeek V4 Flash 0731high8.5%$1.29mini-SWE-agent
33Wonder Fullmax8.0%$28.65mini-SWE-agent
34GPT-6 Lunamax8.0%$0.181mini-SWE-agent
35GPT-5.4xhigh7.0%–mini-SWE-agent
36GLM 5.3 Flashmax7.0%$0.799mini-SWE-agent
37Grok 4.5high4.5%$4.12mini-SWE-agent
38GPT-5.4 Minixhigh3.0%–mini-SWE-agent
39Gemini 3.6 Flashhigh2.5%$5.92mini-SWE-agent
40Muse Spark 1.1xhigh2.5%$13.06mini-SWE-agent
41Muse Spark 1.2xhigh2.5%$2.47mini-SWE-agent
42Sonnet 4.6max2.0%–mini-SWE-agent
43Gemini 3.5 Flashhigh1.5%–mini-SWE-agent
44Qwen 3.6 Plusmax1.0%–mini-SWE-agent
45Qwen 3.8 27Bxhigh1.0%$17.99mini-SWE-agent
46Gemini 3.1 Prohigh1.0%$2.63mini-SWE-agent
47GLM 5.1max1.0%–mini-SWE-agent
48Kimi K2.6max0.5%–mini-SWE-agent
49Kimi K2.7 Codemax0.5%$3.19mini-SWE-agent
50Nemotron 3 Ultra 550B A55Bmax0.5%–mini-SWE-agent
51Inkling Smallxhigh0.5%$0.479mini-SWE-agent
52Haiku 4.5xhigh0.0%–mini-SWE-agent
53DeepSeek V4max0.0%–mini-SWE-agent
54Nemotron 3.5 Lightning–0.0%$0.217mini-SWE-agent
55Gemini 3 Flashhigh0.0%$0.440mini-SWE-agent
56Gemini 3.1 Flash-Litehigh0.0%$0.064mini-SWE-agent
57Gemini 3.5 Flash-Litehigh0.0%$0.337mini-SWE-agent
58Grok 4.3high0.0%$0.356mini-SWE-agent
59MiniMax M2.7high0.0%–mini-SWE-agent
60Laguna M.1–0.0%–mini-SWE-agent
61Laguna XS.2–0.0%–mini-SWE-agent
62Inklingmax0.0%$4.14mini-SWE-agent