Benchmarks / Coding
Real-SWE (Specific Labs)
Resolution rate of coding agents on private, licensed enterprise codebases, published as pass@1 over eight runs per task on a ten-task sample. Maintained by Specific Labs. Score: Resolution rate = pass@1 correctness averaged over eight independent runs per task, with a published 95% confidence interval.
Real-SWE (Specific Labs)
Higher is better · best effort per model
full table below
| # | Model | Effort | Score |
|---|---|---|---|
| 1 | Fable 5.1 | max | 38.8% |
| 2 | GPT-6 Astra | max | 33.8% |
| 3 | Gemini 3.8 Flash | max | 31.3% |
| 4 | GLM 5.3 | max | 28.8% |
| 5 | Muse Spark 1.3 | max | 23.8% |
| 6 | Grok 4.6 | max | 23.8% |
| 7 | Kimi K3 | max | 18.8% |
| 8 | GPT-5.6 Sol | max | 16.3% |