Given four hours and a GPU to improve the way AI is trained, the best agent scored 0.25 out of 1 — and most never tried

A benchmark posted to arXiv on 20 August, AI4AI-Bench, tests something narrower and more interesting than the usual coding evaluations: can an AI agent improve the algorithms used to train AI? The setup is ten frozen research repositories covering ten families of training algorithm. An agent gets four hours on a single B300 GPU to modify the training code; the result is then run for up to twelve hours and scored against hidden benchmarks on a scale where 0 is an uninformative model, 0.1 is the original unmodified algorithm and 1.0 is the best known result. Across 29 configurations of six systems on all ten tasks the mean score was 0.166, and the best single system reached 0.250. The most revealing figure is not about capability at all. Systems that actually attempted a change to the algorithm averaged 0.226 against 0.126 for those that did not — and turning up the reasoning effort raised the share of runs that attempted a change from 8 percent to 64 percent, lifting the mean from 0.094 to 0.196. The paper is a preprint and has not been peer reviewed.

✓ Verified · 2 sources

Related stories