Add a benchmark

Benchmarks / Coding

ProgramBench v1 Fully Resolved

Share of public ProgramBench v1 tasks where the implementation passes every hidden behavioral test.

62models
10%Open Index weight
2026-10-12last updated
ProgramBenchpublisher

ProgramBench v1 Fully Resolved

Higher is better · best effort per model

Chart settings
62 of 62 selected

full table below

#ModelEffortScoreCost / taskHarness
1Opus 5.5max18.5%$68.05mini-SWE-agent
2Fable 5.1max7.0%$58.52mini-SWE-agent
3Sonnet 5.5max6.5%$35.86mini-SWE-agent
4GPT-6 Astramax5.5%$11.57mini-SWE-agent
5Opus 5max3.0%$60.29mini-SWE-agent
6GPT-6.1 Solmax3.0%$2.87mini-SWE-agent
7Haiku 5.5max2.5%$6.83mini-SWE-agent
8Gemini 4high2.5%$17.66mini-SWE-agent
9Muse Spark 1.3max2.5%$54.29mini-SWE-agent
10Fable 5max2.0%$75.68mini-SWE-agent
11Kimi K3max2.0%$70.48mini-SWE-agent
12GPT-6 Solmax2.0%$11.67mini-SWE-agent
13GPT-5.6 Solmax1.5%$15.29mini-SWE-agent
14GLM 5.3max1.5%$21.96mini-SWE-agent
15Opus 4.8max1.0%$31.27mini-SWE-agent
16Gemini 3.8 Flashhigh1.0%$10.93mini-SWE-agent
17Grok 4.6high1.0%$16.76mini-SWE-agent
18Sonnet 4.6max0.5%–mini-SWE-agent
19DeepSeek V4.1 Flashmax0.5%$0.897mini-SWE-agent
20Grok 4.7xhigh0.5%$46.49mini-SWE-agent
21Wonder Fullmax0.5%$28.65mini-SWE-agent
22Muse Spark 1.2xhigh0.5%$2.47mini-SWE-agent
23GPT-5.4high0.5%$2.27mini-SWE-agent
24GPT-5.5xhigh0.5%$6.95mini-SWE-agent
25GPT-5.6 Terramax0.5%$5.38mini-SWE-agent
26GPT-6 Lunamax0.5%$0.181mini-SWE-agent
27Inkling Smallxhigh0.5%$0.479mini-SWE-agent
28MiMo V2.6 Flashhigh0.5%$0.471mini-SWE-agent
29MiMo V2.6 Prohigh0.5%$0.727mini-SWE-agent
30GLM 5.2max0.5%$12.58mini-SWE-agent
31Qwen 3.6 Plusmax0.0%–mini-SWE-agent
32Qwen 3.8 27Bxhigh0.0%$17.99mini-SWE-agent
33Qwen 3.8 Maxmax0.0%$12.36mini-SWE-agent
34Haiku 4.5xhigh0.0%–mini-SWE-agent
35Opus 4.7max0.0%–mini-SWE-agent
36Sonnet 5max0.0%$24.36mini-SWE-agent
37DeepSeek V4 Flash 0731high0.0%$1.29mini-SWE-agent
38DeepSeek V4max0.0%–mini-SWE-agent
39DeepSeek V4 Pro 0813max0.0%$0.432mini-SWE-agent
40Nemotron 3.5 Lightning–0.0%$0.217mini-SWE-agent
41Gemini 3 Flashhigh0.0%$0.440mini-SWE-agent
42Gemini 3.1 Flash-Litehigh0.0%$0.064mini-SWE-agent
43Gemini 3.1 Prohigh0.0%$2.63mini-SWE-agent
44Gemini 3.5 Flashhigh0.0%–mini-SWE-agent
45Gemini 3.5 Flash-Litehigh0.0%$0.337mini-SWE-agent
46Gemini 3.6 Flashhigh0.0%$5.92mini-SWE-agent
47Gemini 3.7 Flashhigh0.0%$7.13mini-SWE-agent
48Grok 4.3high0.0%$0.356mini-SWE-agent
49Grok 4.5high0.0%$4.12mini-SWE-agent
50Kimi K2.6max0.0%–mini-SWE-agent
51Kimi K2.7 Codemax0.0%$3.19mini-SWE-agent
52Muse Spark 1.1xhigh0.0%$13.06mini-SWE-agent
53MiniMax M2.7high0.0%–mini-SWE-agent
54Nemotron 3 Ultra 550B A55Bmax0.0%–mini-SWE-agent
55GPT-5.4 Minixhigh0.0%–mini-SWE-agent
56GPT-5.6 Lunamax0.0%$0.943mini-SWE-agent
57Laguna M.1–0.0%–mini-SWE-agent
58Laguna XS.2–0.0%–mini-SWE-agent
59Hy4–0.0%$12.81mini-SWE-agent
60Inklingmax0.0%$4.14mini-SWE-agent
61GLM 5.1max0.0%–mini-SWE-agent
62GLM 5.3 Flashmax0.0%$0.799mini-SWE-agent