Benchmarks / Long Context
RULER
NVIDIA synthetic benchmark that evaluates long-context LLMs with configurable sequence length and task complexity across 13 tasks in 4 categories. Maintained by NVIDIA. Score: Per-length scores at 4K-128K plus Avg. across the 13 RULER tasks (wAvg. inc/dec also reported); performance above the 4K Llama-2-7b threshold (85.6%) is underlined.
RULER
Higher is better · best effort per model
full table below
| # | Model | Effort | Score |
|---|---|---|---|
| 1 | Gemini 1.5 Pro | none | 95.8% |
| 2 | Qwen 2.5 14B | – | 95.7% |
| 3 | Qwen 3 235B-A22B | xhigh | 95.0% |
| 4 | Qwen 3 14B | none | 94.6% |
| 5 | Qwen 3 32B | none | 93.7% |
| 6 | Qwen 2.5 7B | – | 91.8% |
| 7 | Qwen 3 30B-A3B | none | 91.6% |
| 8 | GPT-4.1106 | – | 91.6% |
| 9 | GLM 4 | – | 89.9% |
| 10 | Llama 3.1 70b | none | 89.6% |
| 11 | Qwen 3 8B | none | 89.1% |
| 12 | Mistral | none | 89.1% |
| 13 | Llama 3.1 8b | medium | 88.3% |
| 14 | Llama 3 70b | none | 86.5% |
| 15 | Mistral Large 2411 | – | 86.0% |
| 16 | Qwen 2 72B | none | 85.9% |
| 17 | Qwen 3 4B | none | 85.2% |
| 18 | Llama 3 8b | none | 82.4% |
| 19 | Mistral Large 2407 | – | 80.5% |
| 20 | GLM 3 | – | 69.6% |
| 21 | Qwen 1.5 72B | – | 55.7% |