Tools
AI Minute Newsroom
2026-08-17
The people who publish the AI leaderboards just released the tool that makes their leaderboards less important
Artificial Analysis, whose model rankings are quoted in most coverage of every new release, announced Optima on 13 August. It is a self-serve service for building a benchmark out of your own material rather than reading someone else's. You give it a dataset, or import real agent traces from tools like Arize, Braintrust or Langfuse, or simply describe your use case in plain language, and it runs candidate models against that and grades them using the same pairwise judging method the company uses for its public benchmarks. Every run reports cost per task and time per task next to the quality score, and your own agent stack can be entered into the same comparison over HTTP. Pricing is usage-based with no subscription. The company frames the goal plainly: find the best model for your task, or an equally good one at roughly a tenth of the cost or time. Testers before launch used it for things like picking a cheap enough model for a finance agent, matching a firm's legal writing style, and sorting a custom image set.
Why it mattersThe honest admission inside this launch is that public leaderboards have stopped predicting production performance — a model that tops an average of academic tests can still be the wrong choice for summarising your support tickets. Artificial Analysis is better placed than anyone to know that, since it runs the leaderboards. For small teams, the useful part is not the grading method but the two columns beside it: cost per task and time per task are what actually decide a deployment, and they are usually discovered after the migration rather than before. Worth remembering that the vendor grading the models here also sells the grading.
✓ Verified · 3 sources
Read in the app — free, in 9 languages
Related stories
OpenRouter dropped three AI providers after a bait-prompt leak test.
2026-11-22Researchers say they broke out of Meta's Muse agent and leaked chats.
2026-11-22ChatGPT's shared workspace Space is now rolling out to Plus users.
2026-11-21Amazon says attackers' AI agents found an unknown flaw on their own.
2026-11-21METR says it can rarely find humans who beat top AI at coding tasks.
2026-11-21