Benchmarks / Long Context
LongBench v2
503 challenging multiple-choice questions with contexts from 8k to 2M words assessing deep understanding and reasoning on realistic long-context multitasks. Maintained by THUDM. Score: Accuracy on 503 multiple-choice questions across six task categories.
LongBench v2
Higher is better · best effort per model
full table below
| # | Model | Effort | Score |
|---|---|---|---|
| 1 | Qwen 3.8 Max | max | 66.3% |
| 2 | Qwen 3.7 Max | max | 65.3% |
| 3 | Gemini 2.5 Pro | high | 63.3% |
| 4 | Gemini 2.5 Flash | medium | 62.1% |
| 5 | Qwen 3 235B-A22B | xhigh | 60.6% |
| 6 | MiMo V2 Flash | high | 60.6% |
| 7 | DeepSeek R1 | xhigh | 58.3% |
| 8 | o1 | high | 57.7% |
| 9 | DeepSeek R1.0528 | – | 56.7% |
| 10 | Gemini 2.0 Flash | medium | 56.0% |
| 11 | GLM 4.5 | xhigh | 50.3% |
| 12 | Qwen 3 30B-A3B | none | 50.1% |
| 13 | Gemini 1206 | – | 49.3% |
| 14 | DeepSeek V3 | none | 48.7% |
| 15 | GLM 4.5 Air | – | 48.6% |
| 16 | GPT-4o | high | 46.0% |
| 17 | Kimi K2 | medium | 44.9% |
| 18 | GLM 4 | – | 44.3% |
| 19 | Qwen 2.5 72B | none | 42.1% |
| 20 | Qwen 3 32B | none | 38.8% |
| 21 | o1 mini | high | 37.8% |
| 22 | Mistral Large 24.11 | – | 34.4% |
| 23 | Llama 3.1 70b | none | 31.6% |
| 24 | Qwen 2.5 7B | – | 30.0% |
| 25 | Llama 3.1 8b | medium | 30.0% |
| 26 | Llama 3.3 70b | none | 29.8% |
| 27 | GPT-4o mini | high | 29.3% |
| 28 | Mistral Large 2 | none | 26.6% |