attention

The central operation in a transformerⓘtransformerruntimeThe neural network architecture that combines self-attention with feed-forward layers, dominant for language modeling since 2017 and the substrate for nearly every modern LLM. Chat about this → Open full entry →. Each token produces three projections (query, key, value) from its hidden state. To compute the output for token i, take the dot product of its query against every prior key, softmax those scores, then take the weighted sum of the corresponding values. That gives the attention output for that position.

Computationally, attention is the part of a transformerⓘtransformerruntimeThe neural network architecture that combines self-attention with feed-forward layers, dominant for language modeling since 2017 and the substrate for nearly every modern LLM. Chat about this → Open full entry → that scales quadratically with sequence length, which is why long-context inferenceⓘinferenceruntimeRunning a trained model to produce outputs (tokens, images, embeddings) from inputs at serving time, as distinct from the gradient updates of training. Chat about this → Open full entry → is hard. Most modern runtime optimizations target attention specifically: FlashAttention reorders the memory access pattern to avoid materializing the full attention matrix, PagedAttention manages the KV cache like virtual memory, and GQA reduces the number of distinct key/value heads to shrink the cache.

Variants worth knowing: multi-head attention (the original), multi-query attention (MQA, one KV per head), grouped-query attentionⓘGQAruntimeAn attention variant where multiple query heads share the same key and value heads, reducing KV cache size with little quality cost compared to full multi-head attention. Chat about this → Open full entry → (GQAⓘGQAruntimeAn attention variant where multiple query heads share the same key and value heads, reducing KV cache size with little quality cost compared to full multi-head attention. Chat about this → Open full entry →, shared KV across head groups), multi-head latent attentionⓘMLAruntimeAn attention variant introduced in DeepSeek-V2 that compresses keys and values through a learned low-rank projection, dramatically shrinking the KV cache. Chat about this → Open full entry → (MLAⓘMLAruntimeAn attention variant introduced in DeepSeek-V2 that compresses keys and values through a learned low-rank projection, dramatically shrinking the KV cache. Chat about this → Open full entry →, used by DeepSeekⓘDeepSeekweightsA Chinese open-weight family known for the V3 MoE base model and the R1 reasoning model, both released under permissive licenses and unusually transparent in their training-cost reporting. Chat about this → Open full entry →-V3), and sliding-window attention (capped lookback for cheaper long-context).