Benchmarks / Coding
DeepSWE v1.1
Pass@1 on 113 original long-horizon software engineering tasks across 91 repos and 5 languages, run with mini-swe-agent in isolated environments (v1.1 fixes dependency drift and flaky tasks).
DeepSWE v1.1
Higher is better · best effort per model
full table below
| # | Model | Effort | Score |
|---|---|---|---|
| 1 | GPT-6 Astra | xhigh | 74.1% |
| 2 | Gemini 3.8 Flash | high | 73.8% |
| 3 | Opus 5 | max | 73.6% |
| 4 | GPT-5.6 Sol | max | 72.7% |
| 5 | Fable 5 | xhigh | 69.9% |
| 6 | GPT-5.6 Terra | max | 69.6% |
| 7 | GLM 5.3 | max | 69.0% |
| 8 | Kimi K3 | max | 68.5% |
| 9 | Grok 4.6 | medium | 67.5% |
| 10 | GPT-5.6 Luna | max | 67.2% |
| 11 | GPT-5.5 | xhigh | 67.0% |
| 12 | Gemini 3.7 Flash | medium | 65.5% |
| 13 | GLM 5.3 Flash | max | 63.4% |
| 14 | DeepSeek V4 Pro | max | 62.8% |
| 15 | Opus 4.8 | max | 59.0% |
| 16 | Qwen 3.8 Max | xhigh | 57.5% |
| 17 | Muse Spark 1.2 | xhigh | 54.9% |
| 18 | Sonnet 5 | max | 53.8% |
| 19 | Grok 4.5 | high | 53.8% |
| 20 | DeepSeek V4 Flash | max | 53.3% |
| 21 | Muse Spark 1.1 | xhigh | 53.3% |
| 22 | GPT-5.4 | xhigh | 51.8% |
| 23 | Gemini 3.6 Flash | high | 46.7% |
| 24 | GLM 5.2 | max | 43.8% |
| 25 | Gemini 3.5 Flash | high | 36.1% |
| 26 | Kimi K2.7 Code | max | 30.5% |
| 27 | Sonnet 4.6 | high | 29.9% |
| 28 | Gemini 3.1 Pro | high | 11.7% |