Benchmarks / Reasoning
HELM Lite
Uses HELM Lite's core-scenarios Accuracy table mean win rate. Score: core-scenarios mean win rate. Collected by Jake Boggs' benchmark dashboard.
HELM Lite
Higher is better · best effort per model
full table below
| # | Model | Effort | Score |
|---|---|---|---|
| 1 | GPT-4o | high | 93.8% |
| 2 | DeepSeek V3 | none | 90.8% |
| 3 | Sonnet 3.5 | medium | 88.5% |
| 4 | GPT-4 | high | 86.7% |
| 5 | Llama 3.1 | max | 85.4% |
| 6 | Gemini 1.5 Pro | none | 84.2% |
| 7 | Llama 3.2 | – | 81.9% |
| 8 | Gemini 2.0 Flash | medium | 81.3% |
| 9 | Llama 3.3 | none | 81.2% |
| 10 | Llama 3 70b | none | 79.3% |
| 11 | Qwen 2 72B | none | 77.0% |
| 12 | Qwen 2.5 Turbo | none | 74.5% |
| 13 | Mistral Large 2 | none | 74.4% |
| 14 | GPT-4o mini | high | 70.1% |
| 15 | Opus 3 | none | 68.3% |
| 16 | Gemma 2 27B | – | 67.5% |
| 17 | Gemini 1.5 Flash | none | 66.7% |
| 18 | Jamba 1.5 Large | none | 63.7% |
| 19 | Qwen 1.5 72B | – | 60.8% |
| 20 | Gemma 2 9B | – | 56.2% |
| 21 | Qwen 1.5 110B | none | 55.0% |
| 22 | Qwen 1.5 32B | – | 54.6% |
| 23 | Haiku 3.5 | medium | 53.1% |
| 24 | Llama 2 70b | none | 48.2% |
| 25 | GPT-3.5 | high | 43.9% |
| 26 | Qwen 1.5 14B | – | 42.5% |
| 27 | Gemini 1.0 Pro | none | 42.2% |
| 28 | Jamba 1.5 Mini | none | 41.4% |
| 29 | Llama 3 8b | none | 38.7% |
| 30 | Sonnet 3 | none | 37.7% |
| 31 | Llama 6 5b | none | 34.5% |
| 32 | Mistral | none | 33.3% |
| 33 | Mistral Large 2402 | – | 32.8% |
| 34 | Command-R (Mar '24) | none | 29.9% |
| 35 | Mistral Small 2402 | – | 28.8% |
| 36 | Qwen 1.5 7B | – | 27.5% |
| 37 | Mistral Medium 2312 | – | 26.8% |
| 38 | Haiku 3 | medium | 26.3% |
| 39 | Llama 2 13b | none | 23.3% |
| 40 | Llama 2 7b | none | 15.2% |