Terminal-Bench v4-AA

AA's 66-task Terminal-Bench v4.0, three repeats, average pass@1. Distinct from the upstream leaderboard and v2.1.

145models

–Open Index weight

–last updated

Artificial Analysispublisher

#	Model	Effort	Score	Cost / task	Harness
1	Sonnet 5.5	max	63.6%	–	mini-SWE-agent
2	Opus 5.5	max	59.6%	–	mini-SWE-agent
3	GPT-6 Astra	xhigh	59.6%	–	mini-SWE-agent
4	Muse Wonder Full (Max)	max	57.1%	–	mini-SWE-agent
5	Gemini 4	high	57.1%	–	mini-SWE-agent
6	GPT-6.1 Sol	max	56.1%	–	mini-SWE-agent
7	Fable 5.1	xhigh	55.1%	–	mini-SWE-agent
8	Muse Wonder	max	52.0%	–	mini-SWE-agent
9	Opus 5	max	49.0%	–	mini-SWE-agent
10	GPT-6 Sol	max	43.9%	–	mini-SWE-agent
11	Fable 5	max	42.4%	–	mini-SWE-agent
12	GLM 5.3	max	41.9%	–	mini-SWE-agent
13	GPT-5.6 Sol	max	39.9%	–	mini-SWE-agent
14	Qwen 3.8 Max	max	38.9%	–	mini-SWE-agent
15	GPT-5.6 Terra	max	35.4%	–	mini-SWE-agent
16	MiMo V2.6 Pro	–	34.8%	–	mini-SWE-agent
17	Muse Spark 1.3	max	33.3%	–	mini-SWE-agent
18	Step 5 Preview	–	33.3%	–	mini-SWE-agent
19	Ling 3.1 Flash	–	33.3%	–	mini-SWE-agent
20	Haiku 5.5	max	32.8%	–	mini-SWE-agent
21	GLM 5.3 Flash	max	32.8%	–	mini-SWE-agent
22	DeepSeek V4.1 Flash	max	26.8%	–	mini-SWE-agent
23	Mistral Large 4	–	26.8%	–	mini-SWE-agent
24	Grok 4.7	xhigh	25.8%	–	mini-SWE-agent
25	Qwen 3.8 Flash	–	25.3%	–	mini-SWE-agent
26	MiMo V2.6 Flash	–	22.7%	–	mini-SWE-agent
27	Opus 4.8	max	21.7%	–	mini-SWE-agent
28	Grok 4.6	high	21.2%	–	mini-SWE-agent
29	Gemini 3.8 Flash	high	19.7%	–	mini-SWE-agent
30	GPT-5.5	xhigh	14.6%	–	mini-SWE-agent
31	DeepSeek V4 Pro 0424	max	14.6%	–	mini-SWE-agent
32	Sonnet 5	max	14.1%	–	mini-SWE-agent
33	DeepSeek V4 Pro 0813	max	14.1%	–	mini-SWE-agent
34	Gemini 3.7 Flash	high	13.6%	–	mini-SWE-agent
35	Kimi K3	max	12.6%	–	mini-SWE-agent
36	GPT-6 Luna	max	12.6%	–	mini-SWE-agent
37	GPT-5.5 Instant	–	12.6%	–	mini-SWE-agent
38	DeepSeek V4 Flash	max	12.1%	–	mini-SWE-agent
39	DeepSeek V4 Flash 0731	max	12.1%	–	mini-SWE-agent
40	GPT-5.6 Luna	max	11.6%	–	mini-SWE-agent
41	Qwen 3.8 2.4T-A95B	–	11.1%	–	mini-SWE-agent
42	Grok 4.5	high	10.6%	–	mini-SWE-agent
43	Muse Spark Open (Xhigh)	xhigh	8.1%	–	mini-SWE-agent
44	Muse Spark 1.2	xhigh	7.1%	–	mini-SWE-agent
45	Gemini 3.6 Flash	high	7.1%	–	mini-SWE-agent
46	Gemini 3.5 Flash	high	6.6%	–	mini-SWE-agent
47	Muse Spark 1.1	xhigh	6.1%	–	mini-SWE-agent
48	Qwen 3.8 27B	xhigh	5.6%	–	mini-SWE-agent
49	Gemini 3.1 Pro	–	4.0%	–	mini-SWE-agent
50	JT-4.1 Flash 236B A21B (Reasoning)	–	3.0%	–	mini-SWE-agent
51	Beam 1.0	–	3.0%	–	mini-SWE-agent
52	Sonnet 4.6	max	3.0%	–	mini-SWE-agent
53	DeepSeek V4 Flash 0420	high	3.0%	–	mini-SWE-agent
54	MiniMax M3	–	2.0%	–	mini-SWE-agent
55	GLM 5.1	–	2.0%	–	mini-SWE-agent
56	GPT-5.4 Mini	xhigh	2.0%	–	mini-SWE-agent
57	K2 Horizon 375B A23B	–	1.5%	–	mini-SWE-agent
58	Qwen 3.7 Max	max	1.5%	–	mini-SWE-agent
59	GLM 5.2	max	1.0%	–	mini-SWE-agent
60	Quasar 438B (Max, Based on GLM-5.2)	max	1.0%	–	mini-SWE-agent
61	Kimi K2.7 Code	–	1.0%	–	mini-SWE-agent
62	Inkling Small	–	1.0%	–	mini-SWE-agent
63	Qwen 3.7 Plus	–	1.0%	–	mini-SWE-agent
64	Inkling	xhigh	1.0%	–	mini-SWE-agent
65	Solar Mini 4	–	1.0%	–	mini-SWE-agent
66	Gemini 3.5 Flash-Lite	high	1.0%	–	mini-SWE-agent
67	K2 Horizon 7B	–	1.0%	–	mini-SWE-agent
68	Kimi K2.6	–	0.5%	–	mini-SWE-agent
69	Hy3	–	0.5%	–	mini-SWE-agent
70	Nemotron 3 Ultra	max	0.5%	–	mini-SWE-agent
71	GPT-5.4 Nano	xhigh	0.5%	–	mini-SWE-agent
72	Muse Glimmer (High)	high	0.5%	–	mini-SWE-agent
73	Ring-2.6-1T	xhigh	0.5%	–	mini-SWE-agent
74	Gemini 3.1 Flash-Lite	high	0.5%	–	mini-SWE-agent
75	Command A	–	0.5%	–	mini-SWE-agent
76	Nemotron 3.5 Lightning	–	0.5%	–	mini-SWE-agent
77	Qwen 3.5 9B	–	0.5%	–	mini-SWE-agent
78	Trinity Large Thinking	–	0.5%	–	mini-SWE-agent
79	North Mini Code	–	0.5%	–	mini-SWE-agent
80	Solar Pro 4	–	0.0%	–	mini-SWE-agent
81	Nemotron 3.5 Super VL	–	0.0%	–	mini-SWE-agent
82	Apodex 1.1	–	0.0%	–	mini-SWE-agent
83	MiMo V2.5 Pro	–	0.0%	–	mini-SWE-agent
84	K2 Horizon MoVA 36B A4B	–	0.0%	–	mini-SWE-agent
85	Grok 4.3	high	0.0%	–	mini-SWE-agent
86	Ling 3.0 Flash	–	0.0%	–	mini-SWE-agent
87	MiniMax M2.7	–	0.0%	–	mini-SWE-agent
88	Ling 3.0 Flash Fin	–	0.0%	–	mini-SWE-agent
89	MiMo V2.5	–	0.0%	–	mini-SWE-agent
90	A.X-K2	–	0.0%	–	mini-SWE-agent
91	Qwen 3.6 27B	–	0.0%	–	mini-SWE-agent
92	Nemotron 3.5 Bolt 74B-A9B	–	0.0%	–	mini-SWE-agent
93	Sonnet 4.5	–	0.0%	–	mini-SWE-agent
94	Ling 3.0 Flash	–	0.0%	–	mini-SWE-agent
95	LongCat 2.0	–	0.0%	–	mini-SWE-agent
96	Qwen 3.5 397B-A17B	–	0.0%	–	mini-SWE-agent
97	Qwen 3.6 35B-A3B	–	0.0%	–	mini-SWE-agent
98	Haiku 4.5	–	0.0%	–	mini-SWE-agent
99	GPT-5 Mini	high	0.0%	–	mini-SWE-agent
100	Gemini 2.5 Pro	–	0.0%	–	mini-SWE-agent
101	K2 Horizon 3.7B	–	0.0%	–	mini-SWE-agent
102	Qwen 3.5 122B-A10B	–	0.0%	–	mini-SWE-agent
103	DeepSeek V3.1	–	0.0%	–	mini-SWE-agent
104	Gemma 4 31B	–	0.0%	–	mini-SWE-agent
105	Mistral Medium 3.5	high	0.0%	–	mini-SWE-agent
106	Nemotron 3 Super 120B A12B (Reasoning)	–	0.0%	–	mini-SWE-agent
107	Qwen 3 235B-A22B	–	0.0%	–	mini-SWE-agent
108	Granite 4.2 30B	–	0.0%	–	mini-SWE-agent
109	MiniCPM5-2B	–	0.0%	–	mini-SWE-agent
110	Mercury 2.5	–	0.0%	–	mini-SWE-agent
111	HyperNova 60B 2605 (High, Based on gpt-oss-120b)	high	0.0%	–	mini-SWE-agent
112	gpt-oss-120b	high	0.0%	–	mini-SWE-agent
113	DeepSeek R1	–	0.0%	–	mini-SWE-agent
114	Mistral Small 4	–	0.0%	–	mini-SWE-agent
115	Granite 4.2 8B	–	0.0%	–	mini-SWE-agent
116	Ling 3.0 Tiny	–	0.0%	–	mini-SWE-agent
117	o3 mini	high	0.0%	–	mini-SWE-agent
118	Mercury 2	high	0.0%	–	mini-SWE-agent
119	Qwen 3 30B-A3B	–	0.0%	–	mini-SWE-agent
120	DeepSeek V3	none	0.0%	–	mini-SWE-agent
121	Llama 4 Maverick	none	0.0%	–	mini-SWE-agent
122	Mistral Large 3	none	0.0%	–	mini-SWE-agent
123	Qwen 3 Coder	none	0.0%	–	mini-SWE-agent
124	Mistral Medium 3.1	none	0.0%	–	mini-SWE-agent
125	Granite 4.2 3B	–	0.0%	–	mini-SWE-agent
126	gpt-oss-20b	high	0.0%	–	mini-SWE-agent
127	Gemma 4 E4B (Reasoning)	–	0.0%	–	mini-SWE-agent
128	NVIDIA Nemotron 3 Nano 30B A3B (Reasoning)	–	0.0%	–	mini-SWE-agent
129	Devstral 2	none	0.0%	–	mini-SWE-agent
130	Solar Pro 3	high	0.0%	–	mini-SWE-agent
131	Devstral Small 2	none	0.0%	–	mini-SWE-agent
132	Qwen 3 32B	–	0.0%	–	mini-SWE-agent
133	Mistral Small 3.1	none	0.0%	–	mini-SWE-agent
134	Mistral Small 3.2	none	0.0%	–	mini-SWE-agent
135	Llama 4 Scout	none	0.0%	–	mini-SWE-agent
136	Qwen 3 14B	–	0.0%	–	mini-SWE-agent
137	Celeris-1	none	0.0%	–	mini-SWE-agent
138	Ministral 3 14B	none	0.0%	–	mini-SWE-agent
139	Ministral 3 8B	none	0.0%	–	mini-SWE-agent
140	Qwen 3 8B	–	0.0%	–	mini-SWE-agent
141	Gemma 3 27B	none	0.0%	–	mini-SWE-agent
142	Ministral 3 3B	none	0.0%	–	mini-SWE-agent
143	NVIDIA Nemotron 3 Nano 4B	–	0.0%	–	mini-SWE-agent
144	Gemma 3 12B	none	0.0%	–	mini-SWE-agent
145	K2 Horizon 0.9B	–	0.0%	–	mini-SWE-agent