MGSM (Vals) · Open Analysis

#	Model	Effort	Score	Cost / task	Harness
1	Opus 4.5	high	95.2%	$0.063	–
2	Opus 4.1	high	94.4%	$0.055	–
3	Sonnet 4.5	high	94.3%	$0.011	–
4	GPT-5.2	xhigh	94.0%	$0.011	–
5	Gemini 3 Pro	high	93.9%	$0.018	–
6	Opus 4	medium	93.8%	$0.027	–
7	o4 mini	high	93.4%	$0.0029	–
8	Gemini 3 Flash	high	93.3%	$0.0071	–
9	Sonnet 4	high	93.0%	$0.0058	–
10	GPT-5.1	high	93.0%	$0.0061	–
11	Sonnet 3.7	medium	93.0%	$0.026	–
12	GPT-5	high	92.8%	$0.014	–
13	Claude 3.5 Sonnet Latest	–	92.6%	$0.0040	–
14	GPT-5 Mini	high	92.6%	$0.0037	–
15	Qwen 3 235B	–	92.5%	$0.0011	–
16	Llama 4 Maverick	medium	92.4%	$0.0002	–
17	DeepSeek R1	xhigh	92.3%	$0.0041	–
18	Haiku 4.5	xhigh	92.1%	$0.0069	–
19	DeepSeek V3	none	92.1%	$0.0004	–
20	Qwen 3 Max	max	92.1%	$0.0023	–
21	gpt-oss-120b	high	92.0%	$0.0007	–
22	o3	high	91.7%	$0.0038	–
23	Grok 3	none	91.3%	$0.0057	–
24	o3 mini	high	91.3%	$0.0082	–
25	Llama 3.3	none	91.1%	$0.0005	–
26	Kimi K2	medium	90.9%	$0.0010	–
27	Grok 4	high	90.9%	$0.026	–
28	Grok 4 Fast	none	90.9%	$0.0006	–
29	Mistral Medium 3.1	none	90.9%	$0.0005	–
30	DeepSeek V3.2	high	90.9%	$0.0001	–
31	GLM 4.5	xhigh	90.8%	$0.0063	–
32	GPT-4o	high	90.7%	$0.0029	–
33	Grok 3 Mini	high	90.4%	$0.0008	–
34	Gemini 2.5 Flash	medium	89.9%	$0.0010	–
35	GLM 4.6	none	89.7%	$0.0043	–
36	Gemini 2.5 Flash Lite	none	89.5%	$0.0002	–
37	Grok 4.1 Fast	high	89.5%	$0.0005	–
38	o1	high	89.3%	$0.057	–
39	GPT-5 Nano	high	89.3%	$0.0019	–
40	Gemini 1.5 Pro	none	89.2%	$0.0010	–
41	Gemini 2.0 Flash	medium	89.0%	$0.0001	–
42	gpt-oss-20b	high	89.0%	$0.0005	–
43	GLM 4.7	none	88.2%	$0.0043	–
44	Llama 4 Scout	medium	88.0%	$0.0001	–
45	MiniMax M2.1	–	87.9%	$0.0013	–
46	GPT-4.1 Mini	high	87.8%	$0.0004	–
47	GPT-4.1	high	87.7%	$0.0017	–
48	Mistral Large 11	–	87.2%	$0.0030	–
49	Gemini 1.5 Flash	none	86.6%	$0.0001	–
50	Magistral Small 1.2	–	86.3%	$0.0015	–
51	GPT-4o mini	high	86.2%	$0.0002	–
52	Grok 2	none	86.1%	$0.0038	–
53	Command A	none	85.7%	$0.0037	–
54	Mistral Large 3	none	85.4%	$0.0005	–
55	Haiku 3.5	medium	84.6%	$0.0011	–
56	Mistral Small 3.1	none	84.2%	$0.0001	–
57	Mistral Small 02	–	84.0%	$0.0002	–
58	Magistral Medium 1.2	medium	74.6%	$0.0057	–
59	Jamba 1.6 Large	none	71.2%	$0.0038	–
60	GPT-4.1 Nano	high	69.3%	$0.0001	–
61	Jamba 1.6 Mini	none	41.7%	$0.0003	–