The people who publish the AI leaderboards just released the tool that makes their leaderboards less important
Artificial Analysis, whose model rankings are quoted in most coverage of every new release, announced Optima on 13 August. It is a self-serve service for building a benchmark out of your own material rather than reading someone else's. You give it a dataset, or import real agent traces from tools like Arize, Braintrust or Langfuse, or simply describe your use case in plain language, and it runs candidate models against that and grades them using the same pairwise judging method the company uses for its public benchmarks. Every run reports cost per task and time per task next to the quality score, and your own agent stack can be entered into the same comparison over HTTP. Pricing is usage-based with no subscription. The company frames the goal plainly: find the best model for your task, or an equally good one at roughly a tenth of the cost or time. Testers before launch used it for things like picking a cheap enough model for a finance agent, matching a firm's legal writing style, and sorting a custom image set.
Related stories
Apple Music will tell you when a song was made by a machine — but the uploader decides whether to say so2026-08-21DeepSeek's cheap workhorse can now see — and on agent tasks that need eyes it says it is close to Anthropic's best
2026-08-21Stripe has just paid $7.5 billion for a model router. Days later Ramp built one and is giving it away until January.
2026-08-21Meta's assistant is now a Mac app that reads your screen and types into any window — and what it sees can train the model
2026-08-21One click on a news site now tells Google to show you more of it — and you stay on the page you were reading
2026-08-21