Kimi Delta Attention: How It Works, Step by Step

Kimi Delta Attention (KDA) is the core linear-attention mechanism introduced in Moonshot AI’s Kimi Linear architecture. Its goal is ambitious: keep the inference efficiency of recurrent or linear attention while recovering enough expressivity to compete with conventional softmax attention.

The key idea is surprisingly compact. Instead of storing every past key and value in a growing KV cache, KDA maintains a fixed-size matrix state that acts like an associative memory. It updates that memory with a delta-rule correction and gives each key-channel its own learned forgetting rate.

That final detail — fine-grained, per-channel forgetting — is what distinguishes KDA from Gated DeltaNet.

This post builds KDA from first principles, starting with ordinary attention, then linear attention, DeltaNet, Gated DeltaNet, and finally KDA itself. We will also look at matrix dimensions, why the delta rule behaves like online learning, how the recurrent state replaces a growing KV cache, why KDA is hardware-friendly, and how Kimi Linear combines KDA with full attention.

1. Why change attention at all?

For a sequence of length $L$, ordinary causal softmax attention computes, schematically,

If $Q,K,V \in \mathbb{R}^{L\times d}$, then $QK^T$ is an $L\times L$ matrix. During training, this gives the familiar quadratic dependence on sequence length.

During autoregressive decoding, implementations avoid recomputing all old keys and values by storing them in a KV cache. But that cache grows linearly with sequence length:

So long-context inference eventually becomes increasingly dominated by memory capacity and memory traffic.

Linear attention asks a different question:

Can the entire history be compressed into a fixed-size state, so each new token reads from and updates that state instead of attending explicitly to every previous token?

That is the family of ideas in which KDA lives.

2. Linear attention as an associative memory

Consider one attention head. For token $t$, let

• $q_t \in \mathbb{R}^{d_k}$ be the query,
• $k_t \in \mathbb{R}^{d_k}$ be the key,
• $v_t \in \mathbb{R}^{d_v}$ be the value,
• $S_t \in \mathbb{R}^{d_k\times d_v}$ be a recurrent memory matrix.

The simplest linear-attention recurrence is

followed by

Check the dimensions:

so it can be added to $S_{t-1}$.

Then

What does the state mean?

Each outer product $k_t v_t^T$ writes an association between a key direction and a value direction into $S$.

A useful mental model is:

$S$ is a tiny learned-at-runtime lookup table represented as a matrix.

Given a query $q_t$, multiplying by $S_t^T$ retrieves the value encoded in directions similar to the query.

Unlike softmax attention, the model no longer stores all previous keys and values separately. The entire past is compressed into $S_t$.

That gives a fixed recurrent-state size of roughly

per head, independent of context length.

But there is an obvious problem.

3. The problem with naive linear attention: memories only accumulate

The update

only adds information. It has no explicit way to correct or erase an old key-value association.

Suppose two different tokens produce similar keys but different values. The state simply accumulates both writes. Over long sequences, unrelated associations interfere with each other.

This is one reason early linear-attention approaches struggled to match softmax attention.

The delta rule fixes this in a particularly elegant way.

4. DeltaNet: do not blindly write — first measure the error

Instead of saying “associate $k_t$ with $v_t$,” ask:

What value does the current memory already predict for $k_t$, and how wrong is it?

The current prediction is

The desired value is $v_t$, so the residual error is

Now update only by that residual:

where $\beta_t\in[0,1]$ is a learned write strength.

Substitute the residual:

Expanding:

Therefore

This is the classical delta rule used by DeltaNet.

Why this is better

The update has two conceptual pieces:

removes the part of memory currently associated with $k_t$, while

writes the new target value.

So the rule performs an erase/correct + write operation rather than blind accumulation.

You can rewrite it in perhaps the most intuitive form:

That reads almost like code:

1. query memory with the current key,
2. calculate prediction error,
3. write only the correction.

5. DeltaNet is online gradient descent

The same equation can be derived from a tiny optimization problem.

Define a reconstruction loss for the current key-value pair:

This says: we would like the memory matrix $S$ to map key $k_t$ to value $v_t$.

Taking one gradient-descent step with learning rate $\beta_t$ gives

The gradient is

Therefore

which is exactly

So an elegant interpretation is:

The attention layer is performing a tiny online learning problem inside the forward pass. The recurrent state is a set of “fast weights,” updated token by token.

The model’s ordinary neural-network weights learn how to produce $q_t$, $k_t$, $v_t$, and the update strength $\beta_t$; the fast state $S_t$ then adapts dynamically to the current sequence.

6. DeltaNet still has a problem: stale memory never globally fades

The delta update can overwrite a memory when a similar key arrives again. But information unrelated to later keys may remain indefinitely.

This motivates Gated DeltaNet (GDN).

GDN adds a learned scalar forget gate $\alpha_t\in[0,1]$:

Now the previous state is multiplied by $\alpha_t$ before being carried forward.

If

memory is preserved.

If

old information decays rapidly.

This is analogous to weight decay on the fast memory.

But notice something coarse about it: one scalar controls the decay of an entire head.

Every row/channel of the state is forgotten at the same rate.

KDA’s main conceptual change is to remove that restriction.

7. Kimi Delta Attention: make forgetting channel-wise

KDA replaces the scalar $\alpha_t$ with a vector

That vector becomes a diagonal matrix:

The KDA recurrence is

and the output remains

This is the central KDA equation.

The critical difference

Gated DeltaNet uses

which means every key dimension decays identically.

KDA uses

so each channel gets an independent retention rate.

One dimension might keep information almost indefinitely:

while another rapidly resets:

This lets the state contain memory components operating at different timescales.

That is much more expressive than asking a whole head to share one memory lifetime.

8. What exactly is being gated?

Because

left-multiplication by

scales the rows of the state.

Those rows correspond to key-space channels.

If we write

then

So KDA can preserve some learned key-space features while aggressively clearing others.

This is why the Kimi Linear paper describes KDA as fine-grained gating.

9. A tiny numerical example

Take a toy state with

Suppose

Let KDA predict the channel-wise retention vector

Then

The first memory channel is mostly preserved; the second is nearly cleared.

Now imagine the current token has

The delta operator is

Applied to the decayed memory:

The new write is

Therefore

Even in this tiny example you can see all three pieces:

1. decay selected memory channels,
2. correct/erase the old association in the current key direction,
3. write the new value.

10. Why does this count as attention?

At first glance KDA looks more like an RNN than attention.

That is partly true: at inference time it is naturally recurrent.

But the state is specifically an associative key-value memory, and the read operation

has the same semantic roles as attention:

• keys determine where information is written,
• values determine what is written,
• queries determine what is retrieved.

The major difference is how history is represented.

Softmax attention

History is explicit:

KDA

History is compressed:

This distinction drives the efficiency difference.

11. Decode complexity: why a fixed state matters

With conventional cached attention, token $t$ compares its query against all previous cached keys. Ignoring implementation details, the amount of attention work and memory traffic therefore grows with context length.

KDA reads and updates a state whose dimensions do not depend on sequence length.

Per head, the persistent recurrent state is

So once decoding is underway, token 10,000 and token 1,000,000 interact with the same-size recurrent state.

This is especially attractive for long-horizon agent workloads, where models may generate or process very long trajectories.

In the Kimi Linear technical report, Moonshot reports up to a 75% KV-cache reduction for its hybrid architecture and up to roughly 6× higher decoding throughput than the full-attention baseline at a 1M-token context. Those are architectural-system results for Kimi Linear as a whole, not a universal constant for every KDA implementation.

12. Why not make the entire model KDA?

A fixed-size recurrent memory is efficient, but it has a fundamental limitation: finite capacity.

Softmax attention can access each previous token individually. A recurrent state must compress all previous information into a fixed number of values.

That makes exact long-range retrieval inherently more difficult.

Kimi Linear handles this pragmatically with a hybrid architecture:

The paper describes a uniform 3:1 ratio of KDA layers to global Multi-Head Latent Attention (MLA) layers.

The KDA layers provide cheap recurrent processing, while periodic global-attention layers restore direct access to the broader context.

This is an important design lesson:

Kimi Linear does not claim that finite-state memory magically eliminates every reason for full attention. It uses full attention sparingly, where its global retrieval capability is valuable.

13. The transition-matrix view

KDA can be written as a state-space recurrence

where

and

Expand $A_t$:

This has the form

which is a specialized case of a Diagonal-Plus-Low-Rank (DPLR) transition.

That structure matters for implementation.

A completely general dense transition matrix would be expressive but expensive. KDA keeps the transition structured enough that chunks of tokens can be processed efficiently using matrix multiplications.

14. Recurrence is great for decode — but training needs parallelism

A token-by-token recurrence appears sequential:

That is ideal during autoregressive decoding, because tokens are generated sequentially anyway.

Training is different. GPUs want large matrix multiplications, not millions of tiny sequential updates.

So KDA uses a chunkwise parallel formulation.

Split a length-$L$ sequence into chunks of $C$ tokens:

Within a chunk, the product of transitions can be compressed into structured terms, while state is passed recurrently between chunks.

Conceptually:

where

• $P_{[j]}$ summarizes how the chunk transforms incoming memory,
• $H_{[j]}$ summarizes the new information written by that chunk.

The Kimi paper derives a compact WY-like representation of the repeated rank-1 updates and then applies a UT transform so much of the work becomes large matrix multiplication.

The important systems point is:

KDA is mathematically recurrent, but its training implementation is reorganized so GPUs can process chunks with high GEMM utilization.

This is similar in spirit to other modern recurrent/linear sequence models: recurrent form for inference, parallel or chunkwise form for training.

15. Why fine-grained decay creates an implementation challenge

A scalar decay is easy to combine across a chunk:

KDA instead has vectors of decay rates:

Cumulative decay becomes element-wise across channels, which complicates the chunkwise algebra and can create numerical issues if implemented naively over long spans.

The Kimi team’s contribution is not only the recurrence itself, but also a specialized hardware-efficient algorithm exploiting the fact that its transition is diagonal plus a tightly structured rank-1 correction.

In the paper’s efficiency analysis, the authors argue that tying the low-rank terms to the key vector lets KDA avoid part of the overhead of a more general DPLR operator.

That implementation detail is easy to overlook, but it is crucial: an asymptotically attractive attention mechanism is not useful if its kernels cannot keep modern GPUs busy.

16. How are $q$, $k$, $v$, $\alpha$, and $\beta$ produced?

In the Kimi Linear model, each head receives token representation $x_t$ and generates the ordinary query/key/value-like vectors plus two gating signals.

The paper uses learned projections along with short convolutions and nonlinearities for $q$, $k$, and $v$. The key and query are normalized, while

is the channel-wise decay vector and

is the write/update strength.

So every token decides two distinct things:

1. What memory should fade?

Controlled by

2. How strongly should I correct/write the current key-value association?

Controlled by

These are separate operations, and that separation is useful conceptually.

A token may decide to preserve most old channels while strongly updating one new association, or rapidly decay selected channels while making only a weak write.

17. KDA versus softmax attention

The tradeoff is therefore not “attention versus no attention.”

It is explicit uncompressed memory versus learned compressed memory.

18. KDA versus DeltaNet and Gated DeltaNet

The progression can be summarized in four equations.

Linear attention

Blindly accumulate associations.

DeltaNet

Correct the old prediction before writing.

Gated DeltaNet

Add one scalar forgetting rate for the whole head.

Kimi Delta Attention

Give every key channel its own forgetting rate.

That is the conceptual ladder from basic linear attention to KDA.

19. The most useful intuition: KDA is a tiny database that learns how to overwrite itself

If you remember only one picture, use this one.

Imagine each head owns a small matrix-shaped database $S$.

For each token:

1. Decay: decide which portions of the database are becoming irrelevant.
2. Probe: use $k_t$ to ask what value is currently stored for this key direction.
3. Compute an error: compare that answer with the new $v_t$.
4. Correct: erase the conflicting part of the old association.
5. Write: store the new association.
6. Read: use $q_t$ to retrieve from the updated database.

In compact form:

Softmax attention keeps the database rows themselves — effectively every prior token’s key and value.

KDA keeps only a learned compressed summary.

20. Why KDA is interesting beyond Kimi

KDA sits at the intersection of several ideas that have recently converged in sequence modeling:

• linear attention — replace explicit pairwise token interactions with a recurrent sufficient state,
• fast weights — let a neural network modify a small temporary memory during its forward pass,
• online learning — interpret state updates as optimization steps on a per-token objective,
• gating — learn how quickly different memories should decay,
• structured state-space transitions — constrain the recurrence so it can be parallelized efficiently,
• hybrid architectures — combine cheap finite-state layers with occasional full/global attention.

KDA is therefore useful not merely as a Kimi-specific trick. It is a concrete example of a broader direction: making the model’s context memory active and editable rather than an ever-growing passive cache.

21. What KDA does not solve

It is worth being precise about the limits.

Finite-state compression is still finite

A $d_k\times d_v$ matrix cannot losslessly encode an arbitrarily long context. Some information will interfere or be forgotten.

Long-context benchmark quality is architecture-dependent

Kimi Linear’s results come from a full trained model with a specific architecture, training recipe, hybrid layer pattern, MoE design, kernels, and system implementation. Dropping the KDA recurrence into an arbitrary model does not guarantee the same gains.

“Linear attention” does not mean every operation is literally $O(L)$ with identical constants

Kernel design, chunk size, precision, head dimensions, batch size, and hardware matter enormously. The point is that the recurrent state avoids explicit growth with the number of past tokens during decode and enables subquadratic sequence processing in the linear-attention layers.

22. Final mental model

The easiest way to distinguish the mechanisms is this:

Softmax attention:

Keep every memory and search over them later.

Linear attention:

Compress every new memory into a matrix.

DeltaNet:

Before writing, correct what the matrix already believes for this key.

Gated DeltaNet:

Also let the entire memory head gradually forget.

Kimi Delta Attention:

Let different memory channels forget at different rates, while preserving the delta-rule overwrite behavior and a transition structure that can be implemented efficiently.

The core equation is only one line:

But that line combines three powerful ideas: learned forgetting, error-correcting memory writes, and fixed-size recurrent state.

That is the essence of Kimi Delta Attention.

Sources and further reading

1. Kimi Team, “Kimi Linear: An Expressive, Efficient Attention Architecture.” Technical report, 2025. https://arxiv.org/abs/2510.26692
2. Moonshot AI — Kimi Linear official repository. https://github.com/MoonshotAI/Kimi-Linear
3. Official KDA kernels in Flash Linear Attention (FLA). https://github.com/fla-org/flash-linear-attention/tree/main/fla/ops/kda

The performance figures in this article are reported by the Kimi Linear authors; readers interested in implementation details and benchmark methodology should consult the technical report directly.