SciCode

SciCode: implementing scientific research code from paper descriptions; sub-problem accuracy.

153models

–Open Index weight

–last updated

Artificial Analysispublisher

#	Model	Effort	Score	Cost / task	Harness
1	Opus 5.5	max	66.9%	$0.469	–
2	Fable 5.1	max	63.1%	$0.659	–
3	Muse Wonder Full (Max)	max	62.6%	–	–
4	Gemini 4	high	61.8%	$0.106	–
5	Sonnet 5.5	max	61.0%	$0.278	–
6	Fable 5	max	61.0%	$0.268	–
7	MiMo V2.6 Pro	–	60.9%	$0.0078	–
8	Muse Wonder	max	60.3%	$0.135	–
9	Gemini 3.7 Flash	medium	59.8%	–	–
10	Muse Spark 1.3	xhigh	59.7%	$0.037	–
11	Kimi K3	max	59.5%	$0.060	–
12	GLM 5.3	max	59.0%	$0.082	–
13	Step 5 Preview	–	58.9%	$0.020	–
14	Muse Spark 1.1	xhigh	58.8%	$0.031	–
15	Gemini 3.1 Pro	–	58.7%	$0.083	–
16	Muse Spark Open (Xhigh)	xhigh	58.4%	$0.040	–
17	GPT-5.6 Sol	high	57.8%	$0.035	–
18	Grok 4.7	high	57.8%	$0.099	–
19	GPT-6 Sol	max	57.6%	$0.042	–
20	Muse Spark 1.2	xhigh	57.4%	$0.060	–
21	Gemini 3.8 Flash	high	56.6%	$0.077	–
22	GPT-6 Astra	max	56.5%	$0.232	–
23	Grok 4.6	high	56.5%	$0.027	–
24	Opus 5	max	56.4%	$0.118	–
25	GPT-5.5	high	56.1%	$0.107	–
26	GPT-6.1 Sol	high	55.8%	$0.013	–
27	Haiku 5.5	max	55.0%	$0.011	–
28	GPT-5.6 Terra	max	55.0%	$0.114	–
29	Grok 4.5	high	55.0%	$0.026	–
30	GPT-6 Luna	max	54.6%	$0.0037	–
31	Opus 4.8	max	54.4%	$0.266	–
32	Sonnet 5	max	54.3%	$0.192	–
33	Mistral Large 4	–	54.2%	$0.067	–
34	Ling 3.1 Flash	–	54.1%	$0.015	–
35	Qwen 3.8 2.4T-A95B	–	54.1%	$0.101	–
36	Gemini 3.5 Flash	high	53.9%	$0.068	–
37	GPT-5.6 Luna	max	53.6%	$0.0093	–
38	Gemini 3.6 Flash	high	53.4%	$0.030	–
39	Qwen 3.8 Max	max	53.2%	$0.111	–
40	GPT-5.5 Instant	–	52.5%	$0.025	–
41	GPT-5.4 Mini	xhigh	52.1%	$0.211	–
42	JT-4.1 Flash 236B A21B (Reasoning)	–	52.0%	–	–
43	DeepSeek V4.1 Flash	max	51.9%	$0.012	–
44	GLM 5.3 Flash	max	51.6%	$0.014	–
45	Kimi K2.6	–	51.5%	$0.043	–
46	MiMo V2.6 Flash	–	51.3%	$0.0028	–
47	GLM 5.2	max	51.2%	$0.055	–
48	DeepSeek V4 Pro 0813	max	51.0%	$0.050	–
49	DeepSeek V4 Pro 0424	max	50.8%	$0.011	–
50	Qwen 3.8 Flash	–	50.6%	$0.013	–
51	MiMo V2.5 Pro	–	50.6%	$0.0049	–
52	DeepSeek V4 Flash 0731	max	50.3%	$0.042	–
53	Sonnet 4.6	max	50.1%	$0.150	–
54	MiniMax M2.7	–	50.1%	–	–
55	DeepSeek V4 Flash	max	49.7%	$0.034	–
56	Inkling Small	–	49.7%	$0.023	–
57	Qwen 3.7 Max	max	49.5%	$0.058	–
58	Agnes 2.5 Pro Beta	–	48.8%	–	–
59	Hy3	–	48.6%	$0.0066	–
60	Beam 1.0	–	48.5%	–	–
61	Grok 4.3	high	48.3%	$0.013	–
62	Quasar 438B (Max, Based on GLM-5.2)	max	48.1%	$0.032	–
63	Solar Pro 4	–	48.0%	$0.0088	–
64	Solar Open2 250B	–	48.0%	–	–
65	Nemotron 3.5 Super VL	–	47.8%	$0.0044	–
66	Kimi K2.7 Code	–	47.8%	$0.024	–
67	Solar Mini 4	–	47.6%	$0.012	–
68	GPT-5.4 Nano	xhigh	47.2%	$0.023	–
69	MiniMax M3	–	47.1%	$0.026	–
70	Inkling	xhigh	47.0%	–	–
71	Qwen 3.8 27B	xhigh	46.6%	$0.064	–
72	Gemini 2.5 Pro	–	46.3%	$0.080	–
73	Qwen 3.7 Plus	–	46.1%	$0.014	–
74	Sonnet 4.5	–	45.7%	$0.129	–
75	Apodex 1.1	–	45.5%	$0.021	–
76	Gemma 4 31B	–	45.5%	$0.0000	–
77	DeepSeek V4 Flash 0420	max	45.3%	$0.0033	–
78	Ring-2.6-1T	xhigh	45.0%	$0.010	–
79	Muse Glimmer (High)	high	44.9%	$0.0063	–
80	GLM 5.1	–	44.8%	$0.044	–
81	Qwen 3.5 397B-A17B	–	44.8%	$0.026	–
82	Ling 3.0 Flash	–	44.2%	–	–
83	MiMo V2.5	–	43.9%	$0.0035	–
84	Step 3.7 Flash	high	43.9%	–	–
85	Gemini 3.1 Flash-Lite	high	43.4%	$0.018	–
86	Nex-N2-Pro	–	43.3%	–	–
87	K2 Horizon 375B A23B	–	42.9%	–	–
88	Qwen 3.6 27B	–	42.8%	$0.020	–
89	o3 mini	high	42.8%	–	–
90	Ling 3.0 Flash Fin	–	42.4%	–	–
91	Motif 3	–	42.2%	–	–
92	Haiku 4.5	–	42.2%	$0.051	–
93	Ling 3.0 Flash	–	42.0%	–	–
94	K-EXAONE 2.0 0803	–	42.0%	–	–
95	Qwen 3 235B-A22B	–	41.4%	–	–
96	Gemini 3.5 Flash-Lite	high	41.3%	$0.013	–
97	A.X-K2	–	41.0%	–	–
98	Trinity Large Thinking	–	40.6%	$0.012	–
99	Nemotron 3 Ultra	max	40.3%	$0.011	–
100	Mistral Medium 3.5	high	40.2%	$0.082	–
101	K2 Horizon MoVA 36B A4B	–	40.0%	–	–
102	Gemma 4 26B	–	40.0%	–	–
103	Nemotron 3.5 Bolt 74B-A9B	–	39.7%	–	–
104	Qwen 3.5 122B-A10B	–	39.7%	$0.016	–
105	GPT-5 Mini	high	39.0%	$0.011	–
106	DeepSeek V3	none	39.0%	$0.0010	–
107	gpt-oss-20b	high	38.9%	$0.0019	–
108	Mistral Small 4	–	38.8%	$0.0051	–
109	North Mini Code	–	38.8%	$0.0000	–
110	Command A	–	38.5%	$0.0000	–
111	Mercury 2.5	–	38.5%	$0.0033	–
112	DeepSeek R1	–	38.3%	$0.028	–
113	DeepSeek V3.1	–	38.0%	–	–
114	Granite 4.2 30B	–	37.8%	$0.0060	–
115	Mercury 2	high	37.7%	$0.0035	–
116	G9v3-39A5B	–	36.8%	–	–
117	Qwen 3.6 35B-A3B	–	36.6%	$0.0095	–
118	Mistral Large 3	none	36.6%	$0.0020	–
119	LongCat 2.0	–	36.3%	$0.049	–
120	Nemotron 3 Super 120B A12B (Reasoning)	–	36.2%	$0.0026	–
121	Qwen 3 Coder	none	36.2%	$0.0031	–
122	Qwen 3 32B	–	36.0%	–	–
123	gpt-oss-120b	high	34.0%	$0.0038	–
124	Qwen 3 30B-A3B	–	33.0%	$0.0075	–
125	Devstral 2	none	32.8%	–	–
126	Mistral Medium 3.1	none	32.4%	–	–
127	Devstral Small 2	none	32.4%	–	–
128	Nemotron 3.5 Lightning	–	32.1%	$0.0032	–
129	Llama 4 Maverick	none	31.7%	$0.0008	–
130	Granite 4.2 8B	–	31.5%	$0.0025	–
131	Qwen 3 14B	–	30.7%	–	–
132	NVIDIA Nemotron 3 Nano 30B A3B (Reasoning)	–	30.6%	$0.0011	–
133	Qwen 3.5 9B	–	29.5%	$0.0018	–
134	Mistral Small 3.2	none	28.6%	$0.0003	–
135	Qwen 3 8B	–	27.9%	–	–
136	Mistral Small 3.1	none	27.8%	$0.0003	–
137	K2 Horizon 7B	–	27.5%	–	–
138	MiniCPM5-2B	–	26.3%	–	–
139	Solar Pro 3	high	25.5%	$0.0069	–
140	Granite 4.2 3B	–	25.3%	$0.0016	–
141	Gemma 4 E4B (Reasoning)	–	24.4%	$0.0003	–
142	Ling 3.0 Tiny	–	24.2%	$0.0000	–
143	Ministral 3 14B	none	23.8%	$0.0007	–
144	Gemma 3 27B	none	23.3%	$0.0000	–
145	K2 Horizon 3.7B	–	22.0%	–	–
146	Celeris-1	none	21.6%	$0.0017	–
147	Llama 4 Scout	none	21.3%	$0.0009	–
148	Ministral 3 8B	none	20.7%	$0.0006	–
149	Gemma 3 12B	none	16.4%	$0.0000	–
150	NVIDIA Nemotron 3 Nano 4B	–	16.1%	–	–
151	Ministral 3 3B	none	15.3%	$0.0005	–
152	LFM2.5-2.6B	–	14.4%	–	–
153	K2 Horizon 0.9B	–	6.9%	–	–