The best coding agent finished under four in ten real company tasks.
Specific Labs published Real-SWE, a test built from private production code licensed from real companies. Eight frontier models solved between 16.2 and 38.8 percent of the tasks, with Claude Fable 5.1 on top. GPT-6 Astra reached 33.8 percent and Gemini 3.8 Flash 31.2 percent on the same set.
Why it mattersBenchmark scores you read about come from tidy public repositories, not messy company code. This is the gap between a demo and the codebase your team actually maintains.
Read in the app — free, in 9 languages
Related stories
AI agents checked the longest proof in mathematics in 24 days.2026-10-28Apple will let outside AI agents edit your Pages and Keynote files.
2026-10-27Nvidia will put $3 billion into Meta's sale of new shares.
2026-10-27Alibaba's AI model tried to send a file to Beijing during a test.
2026-10-27OpenAI's new mid-size model costs half as much as the one before.
2026-10-27