HumanEval · Open Analysis

#	Model	Effort	Score	Cost / task	Harness
1	Kimi K2	medium	94.5%	–	–
2	Sonnet 3.5	medium	93.7%	–	–
3	GPT-5	high	93.4%	–	–
4	Qwen 2.5 Coder	none	92.7%	–	–
5	o1 mini	high	92.4%	–	–
6	Mistral Large 2	none	92.0%	–	–
7	Qwen 2.5	–	91.5%	–	–
8	GPT-4o	high	90.2%	–	–
9	DeepSeek V2.5	none	89.0%	–	–
10	Llama 3.1 405b	none	89.0%	–	–
11	Mistral Small 3.1	none	88.4%	–	–
12	Grok 2	none	88.4%	–	–
13	Llama 3.3 70b	none	88.4%	–	–
14	Qwen 2.5 32B	none	88.4%	–	–
15	Haiku 3.5	medium	88.1%	–	–
16	o1	high	88.1%	–	–
17	GPT-4.5	none	88.0%	–	–
18	Gemma 3 27B	none	87.8%	–	–
19	GPT-4o mini	high	87.2%	–	–
20	GPT-4	high	87.1%	–	–
21	Qwen 2.5 72B	none	86.6%	–	–
22	Qwen 2 72B	none	86.0%	–	–
23	Grok 2 Mini	–	85.7%	–	–
24	Gemma 3 12B	none	85.4%	–	–
25	Opus 3	none	84.9%	–	–
26	Mistral Small 3	none	84.8%	–	–
27	Qwen 2.5 7B	–	84.8%	–	–
28	Gemini 1.5 Pro	none	84.1%	–	–
29	Qwen 2.5 14B	–	83.5%	–	–
30	Codestral	–	81.1%	–	–
31	Llama 3.1 70b	none	80.5%	–	–
32	Qwen 2 7B	–	79.9%	–	–
33	Haiku 3	medium	75.9%	–	–
34	Gemini 1.5 Flash	none	74.3%	–	–
35	Grok 1.5	–	74.1%	–	–
36	Sonnet 3	none	73.0%	–	–
37	Llama 3.1 8b	medium	72.6%	–	–
38	Gemma 3 4B	none	71.3%	–	–
39	GPT-3.5	high	68.0%	–	–
40	Gemma 2 27B	–	51.8%	–	–
41	Gemma 3 1B	none	41.5%	–	–
42	Gemma 2 9B	–	40.2%	–	–