Deriving Kimi Delta Attention from basic linear attention principles

Understanding Complex Attention Variants

Modern linear attention variants, such as Kimi Delta Attention (KDA), often appear complex and difficult to understand at first glance. These variants have evolved over several years, leading to increased complexity that can make them inaccessible to those unfamiliar with their development history.

Derivation Path to KDA

The article outlines a step-by-step derivation process, starting from fundamental softmax attention and progressing through linear attention, DeltaNet, Gated DeltaNet, and finally to KDA. This approach aims to show how these advanced equations can be reached by asserting simple principles about hidden states.

The derivation begins by examining ordinary causal softmax attention, which involves scalar attention weights measuring similarity between keys and queries, followed by a softmax function and a weighted sum of value vectors. The challenge with softmax attention in autoregressive inference is that its denominator depends on the current query and all previous keys, making it difficult to rearrange for efficiency.

Simplifying Attention for Linearization

To simplify the computation and move towards linear attention, the softmax function is removed. By absorbing a constant scale into the query, a bare version of attention is presented, where the scalar inner product can be rearranged. This allows for the collection of all past-dependent elements into a single matrix, a crucial step in developing linear attention mechanisms.

Relevance to Modern AI Models

The DeltaNet family of linear attention variants, including those derived in this explanation, are utilized by contemporary AI models such as the Qwen and Kimi model families. Understanding their underlying mechanics provides insight into the architectural choices of these advanced language models.