Attention residuals could change one of the least questioned components in modern LLMs: the residual pathway. Rather than forcing every Transformer layer to inherit one ever-growing blend of past computation, Kimi’s proposal lets layers retrieve earlier representations selectively.
The idea was popularized in the original video source, which frames it as “attention rotated across depth.” That analogy is useful, but the bigger story for AI builders is practical: this is an attempt to make deeper models more information-efficient without redesigning the core Transformer block.
The bottleneck attention residuals are designed to fix
In a standard pre-norm Transformer, each layer adds its new output to the running residual stream. This shortcut is a major reason very deep networks train reliably, but the Kimi Team argues that it has a cost: previous layer outputs are combined with fixed, equal weight.
As the network gets deeper, the residual stream can grow in magnitude while the distinctive contribution of any single earlier layer becomes harder to preserve. Kimi calls this PreNorm dilution. Put simply, the model keeps carrying a summary of its history forward, but it has no direct, selective mechanism for recovering a useful intermediate representation once it has been mixed into everything else.
That matters because depth is not just a stack of repeated operations. Different layers may encode different useful transformations: syntax, retrieval cues, planning signals, intermediate reasoning features, or code structure. A fixed addition rule assumes each prior update should be treated identically. That is a remarkably rigid assumption for a model that otherwise uses attention to decide which tokens deserve focus.
How attention residuals turn depth into retrieval
Kimi’s Attention Residuals, or AttnRes, replace uniform residual accumulation with softmax attention over earlier layer outputs. At layer l, the model can assign different weights to representations from preceding layers and build its residual input as a weighted combination rather than an indiscriminate sum.
The important distinction is that this is not merely a longer skip connection. The routing is learned and input-dependent. A layer can emphasize a much earlier representation when it is useful, downweight irrelevant stages of computation, and change that choice from one input to another.
The original video’s “90-degree rotation” metaphor captures the intuition well:
- Standard self-attention selects useful information across tokens in a sequence.
- Attention residuals select useful information across layers in the network.
- Standard residuals pass all earlier computation forward through additive accumulation.
- AttnRes turns that path into a selective, weighted retrieval mechanism.
Kimi’s technical report describes the mechanism as a lightweight depth-wise softmax-attention layer. It uses a learned pseudo-query per layer to score prior representations, rather than adding a conventional full attention module with a large set of new projections. That design choice is central to the pitch: AttnRes is intended as a relatively clean, drop-in replacement for standard residual connections, not a wholesale replacement for the Transformer.
Why Block AttnRes makes the idea usable at scale
The obvious problem is cost. If every layer must retain and attend to every prior layer output, full attention residuals create growing memory and communication demands—particularly painful in distributed training, where activations are often recomputed and passed between pipeline stages.
Kimi’s answer is Block Attention Residuals. Instead of making each layer access every prior layer individually, the architecture divides the network into blocks. Layers use ordinary local accumulation within a block, while cross-block retrieval attends to a compact set of block-level summaries.
That changes the systems equation. Full AttnRes requires memory that grows with the number of layers, while Block AttnRes reduces the cross-block memory requirement to the number of blocks. The Kimi repository says a configuration with roughly eight blocks recovers most of the full method’s gains, and reports that Block AttnRes can match a baseline’s loss with 1.25× more training compute.
For founders and infrastructure teams, that is the point worth watching. An architectural idea is only strategically relevant if it survives the realities of activation memory, pipeline parallelism, kernel performance, and training throughput. Kimi combines Block AttnRes with cached pipeline communication and a two-phase computation strategy specifically to keep the overhead marginal rather than theoretical.
The benchmark results are promising—but narrowly scoped
Kimi evaluated AttnRes in its Kimi Linear architecture at 48 billion total parameters and 3 billion activated parameters, trained on 1.4 trillion tokens. In the published results, the method improved every listed downstream benchmark over the baseline.
The largest reported jumps were on tasks where retaining and reusing intermediate computation seems especially valuable. GPQA-Diamond rose from 36.9 to 44.4, Math from 53.5 to 57.1, and HumanEval from 59.1 to 62.2. The team also reports more bounded output magnitudes and a more uniform gradient distribution across depth—evidence consistent with its PreNorm-dilution diagnosis.
Those numbers are compelling, but they should be read as evidence from one research team’s training recipe and model family, not as a settled industry-wide result. There is no public proof yet that the same gains will transfer cleanly to every dense Transformer, mixture-of-experts system, vision-language model, or post-training pipeline.
Still, Kimi has moved beyond a toy demonstration. Its recent Kimi K3 technical blog says the company’s 2.8-trillion-parameter model uses both Kimi Delta Attention and Attention Residuals. That does not independently validate every claim in the paper, but it is meaningful evidence that Moonshot AI considers the residual design production-worthy at a much larger scale.
Attention residuals versus mHC—and the emerging critique
Attention Residuals are part of a broader effort to rethink residual routing. DeepSeek’s manifold-constrained hyper-connections, or mHC, address a related concern from a different direction: instead of retrieving prior states across depth, mHC expands the residual pathway into multiple streams and constrains how those streams mix to preserve stable identity mapping.
The practical difference is straightforward. mHC creates richer parallel residual routes; AttnRes keeps a single broad historical pool and decides which prior depths to retrieve. Neither approach makes the other obsolete. They are alternative answers to the same underlying question: should a deep model’s computational history be handled by fixed addition alone?
There is also an early technical warning for AttnRes itself. A May 2026 preprint on Delta Attention Residuals argues that routing over cumulative hidden states can become redundant in deeper networks, causing attention weights to become too uniform. Its proposed fix is to attend over each layer’s delta—the change introduced by that layer—rather than over cumulative states. The authors report better validation perplexity than both conventional residuals and AttnRes across their tested scales.
This is not a final verdict against Kimi’s approach; it is exactly how fast-moving architecture research should develop. Attention Residuals introduced a useful framing—depth as retrievable memory—and follow-up work is now testing which representations are best to store and retrieve.
What attention residuals mean for AI builders
For most product teams, AttnRes is not something to bolt onto an API workflow next week. It is a pretraining-time architecture decision, with implications for model design, distributed systems, and reproducibility.
But it signals three trends worth tracking:
- Residual connections are becoming an active research frontier. Transformer innovation is no longer limited to token attention, MoE routing, context length, and post-training.
- Efficiency gains may come from better information routing, not just fewer FLOPs. Selectively preserving useful computation can matter as much as making an operation cheaper.
- Reasoning improvements may increasingly be architectural. Reinforcement learning and inference-time search remain important, but better internal access to prior computation could improve the substrate they operate on.
Conclusion: a simple idea with a serious systems test ahead
Attention residuals are compelling because the core proposal is easy to state: do not force every deep layer to inherit the same compressed history; let it retrieve the parts that matter. Kimi’s Block AttnRes makes that principle credible at scale, while its benchmark gains suggest that reasoning-heavy tasks may benefit most.
The open question is whether the approach becomes a standard Transformer ingredient or the first step toward more sophisticated depth-wise memory systems. Either way, Kimi’s work has made residual routing much harder to treat as solved infrastructure.