Benchmarks / Agentic
Toolathlon, original release (HKUST NLP, archived board)
An agent completes 108 real-world tool-use tasks across MCP servers and local toolkits, graded by executing the resulting state against each task's ground truth — the original task definitions, before the June 2026 Verified repair. Maintained by HKUST NLP. Score: Pass@1 in percent as published on the archived pre-Verified board.
Toolathlon, original release (HKUST NLP, archived board)
Higher is better · best effort per model
full table below
| # | Model | Effort | Score |
|---|---|---|---|
| 1 | Gemini 3.5 Flash | xhigh | 56.5% |
| 2 | DeepSeek V4 Pro | max | 52.8% |
| 3 | Opus 4.7 | max | 52.8% |
| 4 | Gemini 3 Flash | high | 49.4% |
| 5 | Gemini 3.1 Pro | high | 48.8% |
| 6 | DeepSeek V4 Flash | max | 48.2% |
| 7 | Sonnet 4.6 | max | 44.8% |
| 8 | GPT-5.2 | xhigh | 43.8% |
| 9 | Opus 4.5 | high | 43.5% |
| 10 | MiniMax M2.1 | – | 40.7% |
| 11 | GLM 5 | high | 39.2% |
| 12 | Sonnet 4.5 | high | 38.9% |
| 13 | GPT-5 | high | 37.7% |
| 14 | Qwen 3.5 Plus | none | 37.7% |
| 15 | GPT-5.1 | high | 37.0% |
| 16 | Gemini 3 Pro | high | 36.4% |
| 17 | DeepSeek V3.2 | high | 35.2% |
| 18 | Sonnet 4 | high | 29.9% |
| 19 | Kimi K2.5 | max | 27.8% |
| 20 | Grok 4 | high | 27.5% |
| 21 | Haiku 4.5 | xhigh | 26.2% |
| 22 | GLM 4.7 | none | 23.8% |
| 23 | GLM 4.6 | none | 18.8% |
| 24 | Grok 4 Fast | none | 18.5% |
| 25 | Kimi K2 | medium | 17.6% |
| 26 | o3 | high | 17.0% |
| 27 | o4 mini | high | 14.8% |
| 28 | GPT-5 Mini | high | 14.5% |
| 29 | Qwen 3 Coder | medium | 14.5% |
| 30 | Gemini 3.1 Flash-Lite | high | 14.2% |
| 31 | Kimi K2.0905 | – | 13.0% |
| 32 | Gemini 2.5 Pro | high | 10.5% |
| 33 | Gemini 2.5 Flash | medium | 3.7% |