Methodology
The Open Index is a weighted average of independent benchmarks. The weights live in open-index.json; changing them takes a pull request and a public discussion.
Components
v0.1| Benchmark | Category | Weight | Models |
|---|---|---|---|
| Terminal-Bench 4.0 | Agentic | 10% | 23 |
| ProgramBench v1 Fully Resolved | Coding | 10% | 62 |
| DeepSWE v1.1 | Coding | 10% | 28 |
| ARC-AGI-2 | Reasoning | 10% | 85 |
| SimpleBench | Reasoning | 10% | 96 |
| BullshitBench v2 | Reasoning | 10% | 108 |
| FrontierMath Tiers 1–3 (v2) | Math | 10% | 84 |
| Finance Agent v2 | Professional | 10% | 77 |
| AutomationBench (Zapier) | Agentic | 10% | 39 |
| GPQA Diamond (Vals) | Knowledge | 10% | 120 |
How a score is computed
- The ranked unit is a model at one reasoning effort. The leaderboard shows each model's best effort; model pages show the rest.
- Scores are used as published. Percentages stay percentages. Elo ratings become win rate against the median model. Nothing is stretched so the leader reads 100.
- Coverage floor. A model is ranked only when measured results cover at least 70% of the index weight.
- Small gaps are estimated. A missing effort is filled from the same model's other efforts, calibrated on benchmarks where both were measured. Remaining gaps come from peers released around the same time. Estimates never count toward the coverage floor.
- Recent models only. Charts show models released in the last three months by default. The filter bar widens that to six months or a year, and the Models panel switches individual models and reasoning efforts on or off. Scores don't change with the filter.
- Cost per task is measured cost to run a task (Artificial Analysis), never token price. It is priced at each lab's subscription plan where one exists, as in Priors; the filter bar switches every cost chart to API list prices.
- Time per task is the speed measure we lead with: how long a task takes end to end, reasoning included (Artificial Analysis). Output speed (tokens per second) and latency are secondary, since fast tokens can still mean a slow answer. Providers and uptime come from OpenRouter.
- Capability indexes average the six best-covered benchmarks in each category, at most two per publisher, with the same rules.
Where the data comes from
Public benchmarks are collected by the open-source scrapers in Priors, which run every two hours. Open Analysis takes that snapshot, and adds every benchmark in community/. Results link to the page they came from.
The engine is shared with Priors, which lets anyone build a custom index with their own weights.