Tool-call error rate

Volume-weighted mean tool-call error rate across OpenRouter providers (3d).

173models

–Open Index weight

–last updated

OpenRouterpublisher

#	Model	Effort	Score	Cost / task	Harness
1	GPT-5.5 Pro	xhigh	0.0%	–	–
2	GPT-5.4 Pro	xhigh	0.0%	–	–
3	GPT-5.2 Pro	xhigh	0.0%	–	–
4	GPT-5 Pro	high	0.0%	–	–
5	o3 pro	high	0.0%	–	–
6	o1	high	0.0%	–	–
7	Mistral Large 2407	–	0.0%	–	–
8	GPT-6 Sol Pro	max	0.0%	–	–
9	GPT-5.2 Codex	xhigh	0.0%	–	–
10	GPT-5.6 Sol Pro	max	0.0%	–	–
11	Grok 4.20	xhigh	0.0%	–	–
12	GPT-6 Astra Pro	max	0.0%	–	–
13	GPT-5.3 Codex	xhigh	0.1%	–	–
14	o4 mini	high	0.1%	–	–
15	GPT-6 Sol	max	0.1%	–	–
16	GPT-5.1 Codex	max	0.1%	–	–
17	GPT-6 Astra	max	0.1%	–	–
18	Grok 4.7	max	0.1%	–	–
19	GLM 4.6	none	0.1%	–	–
20	o3	high	0.1%	–	–
21	GPT-5.2	xhigh	0.1%	–	–
22	Grok 4.5	max	0.1%	–	–
23	GPT-6.1 Sol Pro	–	0.2%	–	–
24	GPT-5 Nano	high	0.2%	–	–
25	Hy4	–	0.2%	–	–
26	Qwen 3.5 Plus	none	0.2%	–	–
27	GPT-5.1 Codex Mini	high	0.2%	–	–
28	GPT-5.6 Luna Pro	–	0.2%	–	–
29	Qwen 3.6 35B-A3B	max	0.2%	–	–
30	GPT-5 Mini	high	0.2%	–	–
31	Grok 4.3	xhigh	0.3%	–	–
32	GPT-5.6 Terra	max	0.3%	–	–
33	Grok 4.6	max	0.3%	–	–
34	GPT-4.1	high	0.3%	–	–
35	GPT-5.1	high	0.4%	–	–
36	Kimi K3	max	0.4%	–	–
37	Opus 4.6	max	0.4%	–	–
38	Mistral Small 4	high	0.4%	–	–
39	GPT-6 Luna	max	0.5%	–	–
40	Muse Spark 1.1	max	0.5%	–	–
41	GLM 4.5 Air	–	0.5%	–	–
42	Qwen 2.5 7B	–	0.5%	–	–
43	Codestral 2508	–	0.6%	–	–
44	Gemini 4	max	0.6%	–	–
45	GLM 4.7 Flash	none	0.6%	–	–
46	GPT-5	high	0.6%	–	–
47	Opus 5.5	max	0.6%	–	–
48	GPT-6 Luna Pro	max	0.6%	–	–
49	Opus 4.5	high	0.6%	–	–
50	Qwen 3.5 27B	high	0.6%	–	–
51	o3 mini	high	0.7%	–	–
52	GPT-5.6 Terra Pro	–	0.7%	–	–
53	Mistral Medium 3	none	0.7%	–	–
54	Fable 5.1	max	0.7%	–	–
55	Mistral Small 3.1	none	0.7%	–	–
56	Opus 5	max	0.8%	–	–
57	GLM 5.2	max	0.8%	–	–
58	GPT-4.1 Mini	high	0.8%	–	–
59	GLM 5.3	max	0.8%	–	–
60	GPT-6.1 Sol	max	0.8%	–	–
61	GPT-5.4 Mini	xhigh	0.8%	–	–
62	GLM 4.5	xhigh	0.8%	–	–
63	GPT-5.6 Luna	max	0.9%	–	–
64	GPT-5.4 Nano	xhigh	0.9%	–	–
65	Grok Build 0.1	–	0.9%	–	–
66	Kimi K2	medium	1.0%	–	–
67	DeepSeek V4.1 Flash	max	1.0%	–	–
68	Gemma 4 26B	xhigh	1.0%	–	–
69	Llama 4 Maverick	medium	1.0%	–	–
70	Hy3	xhigh	1.0%	–	–
71	Qwen 3 235B-A22B	xhigh	1.0%	–	–
72	Fable 5	max	1.0%	–	–
73	GLM 4.5 V	none	1.0%	–	–
74	Opus 4.7	max	1.1%	–	–
75	Kimi K2.6	max	1.1%	–	–
76	Opus 4.1	high	1.1%	–	–
77	Muse Spark 1.2	max	1.2%	–	–
78	Devstral 2	none	1.2%	–	–
79	DeepSeek V3.2	high	1.3%	–	–
80	GPT-4o	high	1.3%	–	–
81	MiniMax M2.5	high	1.3%	–	–
82	Kimi K2.7 Code	max	1.3%	–	–
83	Mistral Medium 3.5	high	1.3%	–	–
84	Qwen 3	high	1.4%	–	–
85	GPT-5.5	xhigh	1.4%	–	–
86	Qwen 3.8 2.4T-A95B	–	1.4%	–	–
87	Haiku 4.5	xhigh	1.4%	–	–
88	Gemini 3.5 Flash	xhigh	1.4%	–	–
89	Sonnet 4.6	max	1.4%	–	–
90	DeepSeek V4 Pro 0813	max	1.5%	–	–
91	Kimi K2.5	max	1.5%	–	–
92	DeepSeek V4 Flash 0731	max	1.6%	–	–
93	GLM 5.3 Flash	max	1.6%	–	–
94	Gemini 3 Flash	high	1.6%	–	–
95	GLM 5	high	1.6%	–	–
96	GPT-4.1 Nano	high	1.6%	–	–
97	Gemini 3.1 Pro	high	1.7%	–	–
98	Mistral Large 3	none	1.9%	–	–
99	Qwen 3 Coder	medium	1.9%	–	–
100	GPT-5.4	xhigh	1.9%	–	–
101	Qwen 3.8 27B	max	2.0%	–	–
102	MiMo V2.6 Flash	high	2.0%	–	–
103	GLM 5 V	high	2.1%	–	–
104	Opus 4.8	max	2.1%	–	–
105	Gemini 3.5 Flash-Lite	xhigh	2.2%	–	–
106	Gemini 3.7 Flash	max	2.3%	–	–
107	Gemma 4 31B	max	2.3%	–	–
108	MiniMax M2.1	–	2.3%	–	–
109	MiniMax M2.7	high	2.3%	–	–
110	Qwen 3 14B	none	2.3%	–	–
111	DeepSeek V4 Flash 0423	–	2.4%	–	–
112	Qwen 3.5 122B-A10B	none	2.4%	–	–
113	DeepSeek V4 Flash	max	2.5%	–	–
114	MiniMax M1	–	2.5%	–	–
115	MiniMax M2	–	2.5%	–	–
116	Qwen 3 30B-A3B	none	2.6%	–	–
117	Llama 3.1 8b	medium	2.6%	–	–
118	Haiku 5.5	max	2.6%	–	–
119	Sonnet 5	max	2.7%	–	–
120	DeepSeek V3.1	none	2.7%	–	–
121	Muse Wonder	max	2.8%	–	–
122	Sonnet 4.5	high	2.8%	–	–
123	MiMo V2.5 Pro	xhigh	2.8%	–	–
124	GPT-4o mini	high	2.8%	–	–
125	Gemini 3.8 Flash	max	2.9%	–	–
126	Sonnet 5.5	max	3.0%	–	–
127	Qwen 3.8 Max	max	3.2%	–	–
128	MiniMax M3	max	3.2%	–	–
129	DeepSeek V3	none	3.2%	–	–
130	Qwen 3.8 Flash	max	3.3%	–	–
131	GLM 5.1	max	3.3%	–	–
132	Gemini 2.5 Flash Lite	none	3.4%	–	–
133	GPT-3.5	high	3.5%	–	–
134	DeepSeek V4 Pro 0423	–	3.6%	–	–
135	Qwen 3.5 35B-A3B	low	3.6%	–	–
136	Qwen 3.5 Flash	max	4.1%	–	–
137	GLM 4.7	none	4.1%	–	–
138	GPT-4	high	4.1%	–	–
139	Mistral Medium 3.1	none	4.2%	–	–
140	gpt-oss-20b	high	4.2%	–	–
141	GPT-5.6 Sol	max	4.3%	–	–
142	Qwen 3.7 Plus	high	4.8%	–	–
143	Qwen 3.5 397B-A17B	max	5.0%	–	–
144	Qwen 2.5 72B	none	5.2%	–	–
145	Gemini 3.6 Flash	xhigh	5.2%	–	–
146	Gemini 2.5 Pro	high	5.3%	–	–
147	Qwen 3 32B	none	5.3%	–	–
148	MiMo V2.6 Pro	high	5.6%	–	–
149	Muse Spark 1.3	max	5.7%	–	–
150	MiMo V2.5	xhigh	5.7%	–	–
151	Inkling Small	xhigh	5.7%	–	–
152	Qwen 3.7 Max	max	5.8%	–	–
153	Mistral	none	5.9%	–	–
154	Gemini 3 Pro	high	6.0%	–	–
155	Inkling	max	6.2%	–	–
156	Gemini 3.1 Flash-Lite	high	6.4%	–	–
157	GLM 4.6 V	low	7.0%	–	–
158	Mistral Large	none	7.2%	–	–
159	Qwen 3.7 Flash	max	7.2%	–	–
160	gpt-oss-120b	high	7.9%	–	–
161	Gemini 2.5 Flash	medium	8.1%	–	–
162	Llama 3.3 70b	none	9.0%	–	–
163	Llama 4 Scout	medium	9.1%	–	–
164	Qwen 3.6 Flash	max	9.3%	–	–
165	Qwen 3.6 Plus	max	9.8%	–	–
166	Mistral Large 4	high	10.7%	–	–
167	Gemma 3 27B	none	10.8%	–	–
168	Qwen 3.5 9B	low	11.0%	–	–
169	Qwen 3.6 27B	max	11.9%	–	–
170	Llama 3.1 70b	none	13.5%	–	–
171	Gemma 3 12B	none	18.4%	–	–
172	Qwen 3.8	xhigh	19.2%	–	–
173	Mistral Small 3.2	none	25.6%	–	–