Benchmarks / Math
BrokenArXiv 06/2026 (MathArena)
Plausible but false proof statements taken from June 2026 arXiv papers; a model scores by refusing to prove them and saying the statement is false as written. Maintained by MathArena. Score: Accuracy (average performance on the competition) with a 95% confidence interval (normal approximation).
BrokenArXiv 06/2026 (MathArena)
Higher is better · best effort per model
full table below
| # | Model | Effort | Score |
|---|---|---|---|
| 1 | Opus 5.5 | high | 100.0% |
| 2 | GPT-6.1 Sol | max | 100.0% |
| 3 | GPT-6 Astra | max | 99.1% |
| 4 | GPT-6 Sol | max | 95.4% |
| 5 | Opus 5 | max | 90.7% |
| 6 | Fable 5.1 | low | 86.6% |
| 7 | GPT-5.5 | xhigh | 69.4% |
| 8 | GPT-5.6 Sol | max | 67.3% |
| 9 | Muse Spark 1.3 | max | 53.2% |
| 10 | Grok 4.7 | xhigh | 52.8% |
| 11 | Kimi K3 | max | 51.9% |
| 12 | Muse Spark 1.1 | max | 50.6% |
| 13 | Fable 5 | max | 47.8% |
| 14 | Qwen 3.8 Max | max | 44.0% |
| 15 | Opus 4.8 | max | 41.7% |
| 16 | DeepSeek V4.1 Flash | max | 39.4% |
| 17 | Grok 4.5 | max | 29.3% |
| 18 | Gemini 3.8 Flash | max | 20.8% |
| 19 | Gemini 3.1 Pro | high | 17.6% |
| 20 | DeepSeek V4 Flash | max | 16.7% |
| 21 | GLM 5.2 | max | 14.2% |
| 22 | Gemini 3.5 Flash | xhigh | 12.3% |
| 23 | Gemini 3.6 Flash | xhigh | 12.0% |
| 24 | Gemini 3.7 Flash | max | 11.1% |
| 25 | Qwen 3.6 35B | – | 3.4% |
| 26 | Qwen 3.5 2B | low | 2.8% |