Benchmarks / Coding
SWE-Bench Pro Public
A software-engineering evaluation of long-horizon tasks in public repositories, scored by patches passing both new and regression tests. Maintained by Scale AI. Score: Resolve rate on the public set.
SWE-Bench Pro Public
Higher is better · best effort per model
full table below
| # | Model | Effort | Score |
|---|---|---|---|
| 1 | Muse Spark 1.1 | max | 61.5% |
| 2 | GPT-5.4 | xhigh | 59.1% |
| 3 | Muse Spark | – | 55.0% |
| 4 | Opus 4.6 | max | 51.9% |
| 5 | Gemini 3.1 Pro | high | 46.1% |
| 6 | Opus 4.5 | high | 45.9% |
| 7 | Gemini 3 Pro | high | 43.3% |
| 8 | GPT-5.2025 | high | 41.8% |
| 9 | GPT-5.2 Codex | xhigh | 41.0% |
| 10 | Qwen 3 Coder | medium | 38.7% |
| 11 | Gemini 3 Flash | high | 34.6% |
| 12 | GPT-5.2 | xhigh | 29.9% |
| 13 | Kimi K2 | medium | 27.7% |
| 14 | Qwen 3 235B-A22B | xhigh | 21.4% |
| 15 | gpt-oss-120b | high | 16.2% |
| 16 | Gemma 3 27B IT | medium | 11.4% |
| 17 | Llama 3.1 405b | none | 11.2% |
| 18 | GLM 4.6 | none | 9.7% |
| 19 | Llama 4 Maverick | medium | 5.2% |
| 20 | Codestral 2405 | – | 1.5% |