Research 2026-08-21

Given four hours and a GPU to improve the way AI is trained, the best agent scored 0.25 out of 1 — and most never tried

A benchmark posted to arXiv on 20 August, AI4AI-Bench, tests something narrower and more interesting than the usual coding evaluations: can an AI agent improve the algorithms used to train AI? The setup is ten frozen research repositories covering ten families of training algorithm. An agent gets four hours on a single B300 GPU to modify the training code; the result is then run for up to twelve hours and scored against hidden benchmarks on a scale where 0 is an uninformative model, 0.1 is the original unmodified algorithm and 1.0 is the best known result. Across 29 configurations of six systems on all ten tasks the mean score was 0.166, and the best single system reached 0.250. The most revealing figure is not about capability at all. Systems that actually attempted a change to the algorithm averaged 0.226 against 0.126 for those that did not — and turning up the reasoning effort raised the share of runs that attempted a change from 8 percent to 64 percent, lifting the mean from 0.094 to 0.196. The paper is a preprint and has not been peer reviewed.

Why it mattersRecursive self-improvement — AI that makes AI better, compounding — is the mechanism behind most fast-takeoff arguments, and it has mostly been argued in the abstract. This is one of the first serious attempts to put a number on it, and the number is small: the best system got a quarter of the way to a result that already exists in the literature. But the failure mode is the part worth remembering. The agents were not mainly failing at the science; most of them were not attempting the science, defaulting to safe tinkering instead — and simply letting them think longer changed that eightfold. That is a limit of habit rather than intelligence, and habits are the kind of thing that get engineered away quickly.

#Coding #AI Agents

✓ Verified · 2 sources

Read in the app — free, in 9 languages

Machine learning read the shape of sick brain cells and picked out nine already-approved drugs that calmed them down
2026-08-21 Rumour: the anonymous model that just topped a coding benchmark, for free, is said to be Zhipu's unreleased flagship
2026-08-21 DeepSeek's cheap workhorse can now see — and on agent tasks that need eyes it says it is close to Anthropic's best
2026-08-21 Nvidia is paying $6 billion for the machine that builds a rival's models — and hiring 109 of the people who ran it
2026-08-21 ChatGPT can now read your iMessages and send them — and the setting that lets it skip asking is the one OpenAI warns about
2026-08-21

Given four hours and a GPU to improve the way AI is trained, the best agent scored 0.25 out of 1 — and most never tried

Related stories