AutomationBench-AA

AutomationBench-AA: building workflow automations; partial-credit score.

147models

–Open Index weight

–last updated

Artificial Analysispublisher

#	Model	Effort	Score	Cost / task	Harness
1	Gemini 4	high	77.5%	$0.609	–
2	Sonnet 5.5	max	71.8%	$1.03	–
3	Muse Wonder Full (Max)	max	71.3%	–	–
4	Opus 5.5	max	69.5%	$1.43	–
5	DeepSeek V4.1 Flash	max	68.9%	$0.173	–
6	GPT-6 Astra	max	68.5%	$1.60	–
7	Muse Wonder	max	68.1%	$0.830	–
8	Grok 4.6	xhigh	67.0%	$0.656	–
9	GPT-6.1 Sol	xhigh	66.6%	$0.251	–
10	Grok 4.7	xhigh	65.6%	$1.08	–
11	MiMo V2.6 Flash	–	64.1%	$0.017	–
12	GLM 5.3	max	62.2%	$0.460	–
13	Gemini 3.7 Flash	high	62.0%	$0.294	–
14	Ling 3.1 Flash	–	61.7%	$0.094	–
15	GPT-6 Sol	xhigh	61.7%	$0.292	–
16	Gemini 3.8 Flash	medium	60.9%	$0.354	–
17	GLM 5.3 Flash	max	60.4%	$0.032	–
18	GPT-5.6 Sol	max	60.1%	$0.972	–
19	Mistral Large 4	–	59.9%	$0.313	–
20	GPT-5.6 Terra	max	59.6%	$0.730	–
21	Fable 5.1	max	59.4%	$2.61	–
22	MiMo V2.6 Pro	–	58.6%	$0.042	–
23	Kimi K3	max	58.3%	$0.633	–
24	Grok 4.5	high	57.9%	$0.473	–
25	Muse Spark 1.3	max	57.9%	$0.363	–
26	Qwen 3.8 2.4T-A95B	–	57.2%	$0.353	–
27	DeepSeek V4 Pro 0813	max	56.7%	$0.341	–
28	Opus 5	max	56.6%	$1.32	–
29	DeepSeek V4 Pro 0424	max	56.3%	$0.084	–
30	Qwen 3.8 Max	max	56.2%	$0.398	–
31	Qwen 3.8 Flash	–	55.9%	$0.027	–
32	Fable 5	max	54.1%	$2.80	–
33	DeepSeek V4 Flash 0731	max	54.0%	$0.105	–
34	GPT-6 Luna	max	53.2%	$0.036	–
35	Gemini 3.6 Flash	high	53.0%	$0.464	–
36	Step 5 Preview	–	51.0%	$0.225	–
37	GPT-5.6 Luna	max	50.2%	$0.091	–
38	Muse Spark Open (Xhigh)	xhigh	49.7%	$0.463	–
39	Beam 1.0	–	48.5%	–	–
40	Qwen 3.8 27B	xhigh	48.2%	$0.254	–
41	DeepSeek V4 Flash	max	47.5%	$0.104	–
42	GPT-5.5	xhigh	47.3%	$2.33	–
43	JT-4.1 Flash 236B A21B (Reasoning)	–	46.5%	–	–
44	Opus 4.8	max	45.6%	$2.48	–
45	Gemini 3.5 Flash	high	42.1%	$1.42	–
46	Muse Spark 1.2	xhigh	40.6%	$0.512	–
47	Muse Spark 1.1	xhigh	38.8%	$0.754	–
48	K2 Horizon 375B A23B	–	37.2%	–	–
49	Sonnet 5	max	36.5%	$1.00	–
50	GPT-5.5 Instant	–	36.4%	$0.963	–
51	Haiku 5.5	xhigh	36.0%	$0.028	–
52	Gemini 3.1 Pro	–	35.4%	$0.745	–
53	DeepSeek V4 Flash 0420	high	28.5%	$0.050	–
54	GLM 5.2	max	28.4%	$0.770	–
55	Apodex 1.1	–	28.2%	$0.227	–
56	GPT-5.4 Mini	xhigh	26.2%	$0.663	–
57	K2 Horizon MoVA 36B A4B	–	26.1%	–	–
58	Gemini 3.5 Flash-Lite	high	25.0%	$0.218	–
59	Kimi K2.7 Code	–	24.5%	$0.351	–
60	Quasar 438B (Max, Based on GLM-5.2)	max	23.9%	$0.739	–
61	Qwen 3.7 Max	max	23.0%	$1.26	–
62	Solar Mini 4	–	22.3%	$0.180	–
63	MiniMax M3	–	21.3%	$0.115	–
64	K2 Horizon 7B	–	20.9%	–	–
65	GLM 5.1	–	20.3%	$0.554	–
66	Sonnet 4.6	max	20.1%	$1.18	–
67	MiMo V2.5	–	18.4%	$0.018	–
68	Hy3	–	18.4%	$0.052	–
69	Nex-N2-Pro	–	17.6%	–	–
70	Qwen 3.7 Plus	–	17.4%	$0.194	–
71	Ling 3.0 Flash	–	15.7%	–	–
72	Nemotron 3.5 Bolt 74B-A9B	–	15.5%	–	–
73	Nemotron 3.5 Super VL	–	14.8%	$0.166	–
74	K2 Horizon 3.7B	–	14.7%	–	–
75	Sonnet 4.5	–	14.1%	$0.626	–
76	MiMo V2.5 Pro	–	13.7%	$0.040	–
77	Solar Pro 4	–	13.6%	$0.188	–
78	Kimi K2.6	–	13.0%	$0.610	–
79	Qwen 3.6 27B	–	11.0%	$0.379	–
80	Ling 3.0 Flash Fin	–	7.4%	–	–
81	Qwen 3.5 397B-A17B	–	6.9%	$0.487	–
82	Muse Glimmer (High)	high	6.8%	$0.080	–
83	Gemini 3.1 Flash-Lite	high	6.8%	$0.080	–
84	GPT-5 Mini	high	6.5%	$0.075	–
85	Mistral Medium 3.5	high	6.3%	$0.530	–
86	GPT-5.4 Nano	xhigh	6.0%	$0.033	–
87	Qwen 3.5 122B-A10B	–	5.5%	$0.238	–
88	Qwen 3.6 35B-A3B	–	5.2%	$0.385	–
89	Inkling	xhigh	5.0%	–	–
90	Gemma 4 31B	–	4.9%	$0.0000	–
91	Inkling Small	–	4.8%	$0.057	–
92	LongCat 2.0	–	4.5%	$0.129	–
93	MiniMax M2.7	–	4.4%	–	–
94	Mistral Medium 3.1	none	4.1%	–	–
95	Nemotron 3 Super 120B A12B (Reasoning)	–	3.8%	$0.226	–
96	Haiku 4.5	–	3.2%	$0.108	–
97	Ling 3.0 Flash	–	3.2%	–	–
98	Devstral 2	none	3.1%	–	–
99	Devstral Small 2	none	3.1%	–	–
100	Nemotron 3 Ultra	max	3.0%	$0.214	–
101	DeepSeek V3.1	–	2.7%	–	–
102	A.X-K2	–	2.5%	–	–
103	Mercury 2.5	–	2.4%	$0.185	–
104	Gemini 2.5 Pro	–	2.2%	$0.177	–
105	Qwen 3 235B-A22B	–	2.2%	–	–
106	Granite 4.2 8B	–	2.1%	$0.022	–
107	Gemma 4 26B	–	1.9%	–	–
108	Command A	–	1.8%	$0.0000	–
109	MiniCPM5-2B	–	1.8%	–	–
110	Mercury 2	high	1.7%	$0.051	–
111	Granite 4.2 30B	–	1.5%	$0.041	–
112	Mistral Large 3	none	1.5%	$0.020	–
113	North Mini Code	–	1.5%	$0.0000	–
114	Grok 4.3	high	1.4%	$0.042	–
115	Trinity Large Thinking	–	1.3%	$0.049	–
116	Mistral Small 3.1	none	1.2%	$0.012	–
117	Qwen 3 Coder	none	1.1%	$0.120	–
118	Qwen 3.5 9B	–	1.0%	$0.094	–
119	NVIDIA Nemotron 3 Nano 30B A3B (Reasoning)	–	1.0%	$0.047	–
120	Qwen 3 32B	–	1.0%	–	–
121	Mistral Small 4	–	1.0%	$0.0095	–
122	Nemotron 3.5 Lightning	–	0.8%	$0.028	–
123	Qwen 3 30B-A3B	–	0.8%	$0.033	–
124	Ring-2.6-1T	xhigh	0.7%	$0.232	–
125	Ministral 3 3B	none	0.7%	$0.0029	–
126	DeepSeek V3	none	0.6%	$0.013	–
127	Ministral 3 14B	none	0.6%	$0.027	–
128	o3 mini	high	0.5%	–	–
129	Ministral 3 8B	none	0.5%	$0.025	–
130	Solar Pro 3	high	0.5%	$0.055	–
131	Gemma 4 E4B (Reasoning)	–	0.5%	$0.0008	–
132	NVIDIA Nemotron 3 Nano 4B	–	0.4%	–	–
133	Celeris-1	none	0.4%	$0.018	–
134	Granite 4.2 3B	–	0.4%	$0.0023	–
135	Qwen 3 8B	–	0.3%	–	–
136	Mistral Small 3.2	none	0.3%	$0.0013	–
137	Ling 3.0 Tiny	–	0.3%	$0.0000	–
138	DeepSeek R1	–	0.2%	$0.038	–
139	Llama 4 Maverick	none	0.2%	$0.0031	–
140	K2 Horizon 0.9B	–	0.2%	–	–
141	gpt-oss-120b	high	0.2%	$0.0032	–
142	gpt-oss-20b	high	0.2%	$0.0017	–
143	Magistral Small 1.2	–	0.2%	–	–
144	Llama 4 Scout	none	0.2%	$0.0002	–
145	Qwen 3 14B	–	0.2%	–	–
146	Gemma 3 27B	none	0.2%	$0.0000	–
147	Gemma 3 12B	none	0.2%	$0.0000	–