When a small model hands a conversation to a big one, the big one re-reads everything. Nvidia says a straight line can do the translation instead.

Systems that start cheap and escalate to a stronger model pay a hidden tax: the receiving model re-reads the entire conversation from scratch, the step called prefill, which is where most of the cost of a long prompt sits. In a paper on arXiv (2608.03893) that drew wider attention this week, Nvidia researchers report that the internal key-value cache of one model in a family maps onto another's through ordinary linear regression — fitted without gradient descent, from a calibration set of just 500 sequences of 1,024 tokens, after stripping positional encoding so the fit works at any context length. Transferring a 32,768-token cache took 278 milliseconds against roughly 7 seconds to re-prefill, and across six model pairs in three families the mapping ran 2.7 to 25 times faster than re-reading. The honest part is in the results: it kept 73 to 98 percent of the receiving model's own accuracy on four of the six pairs, and two degraded sharply.

✓ Verified · 2 sources

Related stories