Add a benchmark

Benchmarks / Coding

ProgramBench v1 Raw Pass Rate

Average percentage of hidden behavioral tests passed per public ProgramBench v1 task.

62models
–Open Index weight
2026-10-12last updated
ProgramBenchpublisher

ProgramBench v1 Raw Pass Rate

Higher is better · best effort per model

Chart settings
62 of 62 selected

full table below

#ModelEffortScoreCost / taskHarness
1Opus 5.5max87.0%$68.05mini-SWE-agent
2GPT-6 Astramax85.4%$11.57mini-SWE-agent
3GPT-6.1 Solmax85.3%$2.87mini-SWE-agent
4Gemini 4high83.7%$17.66mini-SWE-agent
5Fable 5.1max82.7%$58.52mini-SWE-agent
6Opus 5max82.3%$60.29mini-SWE-agent
7GPT-6 Solmax81.9%$11.67mini-SWE-agent
8Haiku 5.5max78.7%$6.83mini-SWE-agent
9GPT-5.6 Solmax77.6%$15.29mini-SWE-agent
10Fable 5max76.8%$75.68mini-SWE-agent
11Sonnet 5.5max76.3%$35.86mini-SWE-agent
12Grok 4.7xhigh75.7%$46.49mini-SWE-agent
13Grok 4.6high74.0%$16.76mini-SWE-agent
14GPT-5.6 Terramax72.3%$5.38mini-SWE-agent
15Sonnet 5max72.1%$24.36mini-SWE-agent
16DeepSeek V4.1 Flashmax72.0%$0.897mini-SWE-agent
17Opus 4.8max71.9%$31.27mini-SWE-agent
18Gemini 3.8 Flashhigh71.9%$10.93mini-SWE-agent
19Muse Spark 1.3xhigh71.7%$11.56mini-SWE-agent
20MiMo V2.6 Flashhigh71.7%$0.471mini-SWE-agent
21GPT-5.5xhigh70.8%$6.95mini-SWE-agent
22DeepSeek V4 Pro 0813max70.1%$0.432mini-SWE-agent
23MiMo V2.6 Prohigh69.8%$0.727mini-SWE-agent
24Hy4–69.2%$12.81mini-SWE-agent
25Gemini 3.7 Flashhigh68.7%$7.13mini-SWE-agent
26GPT-5.4xhigh68.6%–mini-SWE-agent
27GPT-5.6 Lunamax68.3%$0.943mini-SWE-agent
28GLM 5.3max66.3%$21.96mini-SWE-agent
29GPT-6 Lunamax66.2%$0.181mini-SWE-agent
30Grok 4.5high65.2%$4.12mini-SWE-agent
31Opus 4.7max64.2%–mini-SWE-agent
32Kimi K3max62.8%$70.48mini-SWE-agent
33GLM 5.2max62.6%$12.58mini-SWE-agent
34Muse Spark 1.2xhigh61.9%$2.47mini-SWE-agent
35Gemini 3.6 Flashhigh59.9%$5.92mini-SWE-agent
36Sonnet 4.6max59.1%–mini-SWE-agent
37DeepSeek V4 Flash 0731high57.3%$1.29mini-SWE-agent
38Gemini 3.5 Flashhigh56.4%–mini-SWE-agent
39GPT-5.4 Minixhigh54.4%–mini-SWE-agent
40GLM 5.1max50.9%–mini-SWE-agent
41Kimi K2.7 Codemax49.4%$3.19mini-SWE-agent
42GLM 5.3 Flashmax48.6%$0.799mini-SWE-agent
43Kimi K2.6max48.0%–mini-SWE-agent
44DeepSeek V4max47.8%–mini-SWE-agent
45Inkling Smallxhigh42.5%$0.479mini-SWE-agent
46Qwen 3.8 Maxmax40.0%$12.36mini-SWE-agent
47Gemini 3.1 Prohigh39.5%$2.63mini-SWE-agent
48Gemini 3.5 Flash-Litehigh33.9%$0.337mini-SWE-agent
49Gemini 3 Flashhigh33.8%$0.440mini-SWE-agent
50Qwen 3.6 Plusmax28.6%–mini-SWE-agent
51Muse Spark 1.1xhigh27.1%$13.06mini-SWE-agent
52Haiku 4.5xhigh25.4%–mini-SWE-agent
53Inklingmax24.4%$4.14mini-SWE-agent
54Wonder Fullmax24.2%$28.65mini-SWE-agent
55Grok 4.3high22.1%$0.356mini-SWE-agent
56Nemotron 3.5 Lightning–18.2%$0.217mini-SWE-agent
57Laguna XS.2–17.1%–mini-SWE-agent
58Laguna M.1–15.8%–mini-SWE-agent
59Gemini 3.1 Flash-Litehigh14.3%$0.064mini-SWE-agent
60MiniMax M2.7high13.5%–mini-SWE-agent
61Qwen 3.8 27Bxhigh11.2%$17.99mini-SWE-agent
62Nemotron 3 Ultra 550B A55Bmax10.0%–mini-SWE-agent