Add a benchmark

Benchmarks / Coding

SWE-bench Verified

A 500-instance human-filtered subset of SWE-bench created with OpenAI, served as the default Verified leaderboard view. Maintained by SWE-bench Team. Score: % Resolved: percentage of task instances solved.

59models
–Open Index weight
–last updated

SWE-bench Verified

Higher is better · best effort per model

Chart settings
59 of 59 selected

full table below

#ModelEffortScoreCost / taskHarness
1Opus 4.8max88.6%––
2Opus 4.7max87.6%––
3Nex-N2-Pro–80.8%––
4Kimi K2.6max80.2%–in-house evaluation framework adapted from SWE-agent (minimal tools: bash, createfile, insert, view, strreplace, submit)
5Gemini 3 Prohigh77.4%–live-SWE-agent
6Qwen 3.6 27Bmax77.2%–internal agent scaffold (bash + file-edit tools)
7Qwen 3.5 397B-A17Bmax76.2%–internal agent scaffold (bash + file-edit tools)
8Gemini 3 Flashhigh75.8%–mini-SWE-agent
9MiniMax M2.5high75.8%–mini-SWE-agent
10Opus 4.6max75.6%–mini-SWE-agent
11Opus 4medium75.2%–TRAE
12Qwen 3.5 27Bhigh75.0%–internal agent scaffold (bash + file-edit tools)
13GPT-4.1high74.6%–JoyCode
14o4 minihigh74.4%–Refact.ai Agent
15GPT-5high74.4%–Prometheus-v1.2.1
16Qwen 3.6 35B-A3Bmax73.4%–internal agent scaffold (bash + file-edit tools)
17MiMo V2 Flashhigh73.4%––
18Opus 2025.05–73.2%–Tools
19GLM 5high72.8%–mini-SWE-agent
20GPT-5.2high72.8%–mini-SWE-agent
21GPT-5.2 Codexxhigh72.8%–mini-SWE-agent
22Sonnet 2025.05–72.4%–Tools
23Devstral 2none72.2%––
24Kimi K2medium71.2%–Lingxi v1.5
25Kimi K2.5high70.8%–mini-SWE-agent
26DeepSeek V3.2high70.0%–mini-SWE-agent
27Qwen 3 Codermedium69.6%–OpenHands
28GLM 4.6none68.2%–Undisclosed
29Devstral Small 2none68.0%––
30GPT-5.1medium66.0%–mini-SWE-agent
31GPT-5.1 Codexmedium66.0%–mini-SWE-agent
32o1high64.6%–W&B Programmer O1 crosscheck5
33GLM 4.5xhigh64.2%–Undisclosed
34Sonnet 2025.02–63.2%–Tools
35MiniMax M2–61.0%–mini-SWE-agent
36GPT-5 Minimedium59.8%–mini-SWE-agent
37o3high58.4%–mini-SWE-agent
38Devstral Small 2512–56.4%–mini-SWE-agent
39Devstral 2512–53.8%–mini-SWE-agent
40Gemini 2.5 Prohigh53.6%–mini-SWE-agent
41Gemini 2.0 Flashmedium52.2%–Google Jules
42Sonnet 2024.10–49.0%–Tools
43Devstral Small 2505–46.8%–OpenHands
44o3 minihigh42.4%–Agentless Lite
45DeepSeek V3none42.0%–agentless framework (Xia et al., 2024)
46Llama 3–41.2%–Agentless Mini
47Qwen 2.5 72Bnone40.6%–Nebius AI
48GPT-4ohigh38.8%–Agentless-1.5
49Devstral Small 2507–38.0%–SWE-agent
50GPT-5 Nanomedium34.8%–mini-SWE-agent
51Qwen 2.5 7B–32.8%–SWE-Fixer
52Gemini 2.5 Flashmedium28.7%–mini-SWE-agent
53gpt-oss-120bhigh26.0%–mini-SWE-agent
54GPT-4.1 Minihigh23.9%–mini-SWE-agent
55GPT-4high22.4%–SWE-agent
56Llama 4 Maverickmedium21.0%–mini-SWE-agent
57Llama 4 Scoutmedium9.1%–mini-SWE-agent
58Qwen 2.5 Codernone9.0%–mini-SWE-agent
59Llama 1 3b–1.2%–RAG baseline