The people who publish the AI leaderboards just released the tool that makes their leaderboards less important

Artificial Analysis, whose model rankings are quoted in most coverage of every new release, announced Optima on 13 August. It is a self-serve service for building a benchmark out of your own material rather than reading someone else's. You give it a dataset, or import real agent traces from tools like Arize, Braintrust or Langfuse, or simply describe your use case in plain language, and it runs candidate models against that and grades them using the same pairwise judging method the company uses for its public benchmarks. Every run reports cost per task and time per task next to the quality score, and your own agent stack can be entered into the same comparison over HTTP. Pricing is usage-based with no subscription. The company frames the goal plainly: find the best model for your task, or an equally good one at roughly a tenth of the cost or time. Testers before launch used it for things like picking a cheap enough model for a finance agent, matching a firm's legal writing style, and sorting a custom image set.

✓ Verified · 3 sources

Related stories