Benchmarks / Agentic
VitaBench
Can an agent finish a real-life errand that spans food delivery, restaurant bookings and travel — 66 tools, a customer whose wishes shift mid-conversation, and no domain policy to lean on? Maintained by Meituan LongCat team. Score: Cross-Scenarios Avg @4 on the 100 cross-scenario tasks (the paper's main result), graded by the benchmark's rubric-based sliding-window LLM evaluator; the page does not define @4 in words — by the column name it is the average over four trials.
VitaBench
Higher is better · best effort per model
full table below
| # | Model | Effort | Score |
|---|---|---|---|
| 1 | Gemini 3 Flash | high | 32.5% |
| 2 | Gemini 3 Pro | high | 31.5% |
| 3 | o3 | high | 26.3% |
| 4 | GPT-5.2 | xhigh | 24.3% |
| 5 | DeepSeek V3.2 | high | 24.0% |
| 6 | o4 mini | high | 19.5% |
| 7 | GLM 4.7 | none | 18.3% |
| 8 | Qwen 3 235B-A22B | xhigh | 14.5% |
| 9 | Qwen 3 Max | max | 14.3% |
| 10 | Kimi K2 | medium | 12.8% |
| 11 | Kimi K2.0905 | – | 11.5% |
| 12 | Qwen 3 32B | none | 5.3% |