Benchmarks / Reasoning
CharXiv Reasoning
Can a multimodal model reason over the real, messy charts in scientific papers — combining values, trends and labels across a figure — rather than read off templated plots? Maintained by Princeton Language and Intelligence. Score: Reasoning accuracy in percent on the validation split: the share of its 1,000 reasoning questions (one per chart) answered correctly, each final answer extracted and scored 0/1 against the ground truth by GPT-4o.
CharXiv Reasoning
Higher is better · best effort per model
full table below
| # | Model | Effort | Score |
|---|---|---|---|
| 1 | o3 | high | 78.6% |
| 2 | o4 mini | high | 72.0% |
| 3 | Sonnet 3.7 | medium | 64.2% |
| 4 | GPT-4.1 Mini | high | 56.8% |
| 5 | GPT-4.1 | high | 56.7% |
| 6 | GPT-4.5 | none | 55.4% |
| 7 | o1 | high | 52.6% |
| 8 | Qwen 2.5 | – | 49.7% |
| 9 | GPT-4o | high | 47.1% |
| 10 | Gemini 1.5 Pro | none | 43.3% |
| 11 | GPT-4.1 Nano | high | 40.5% |
| 12 | GPT-4o mini | high | 34.1% |
| 13 | Gemini 1.5 Flash | none | 33.9% |
| 14 | Sonnet 3 | none | 32.2% |
| 15 | Haiku 3 | medium | 31.8% |
| 16 | Llama 3.2 | – | 31.2% |
| 17 | Opus 3 | none | 30.2% |
| 18 | GLM 4 V | – | 29.1% |
| 19 | Gemma 2 9B | – | 28.4% |
| 20 | Qwen 2 | – | 24.5% |
| 21 | Gemini 1.0 Pro | none | 22.8% |
| 22 | Llama 3 8b | none | 19.0% |
| 23 | Mistral | none | 13.9% |
| 24 | Llama 2 | – | 8.8% |