Watch attention turn alignment into probability.

A transformer scores query-key pairs, scales the logits, masks forbidden positions, and applies softmax. Change each stage and see where probability mass moves.

Open the attention field
QUERY MEETS KEY · DOT PRODUCT BECOMES LOGIT · SCALE CONTROLS SATURATION · MASK REMOVES FUTURE TOKENS · SOFTMAX NORMALIZES · VALUES ARE MIXED · QUERY MEETS KEY · DOT PRODUCT BECOMES LOGIT · SCALE CONTROLS SATURATION · MASK REMOVES FUTURE TOKENS ·
42%peak attention
1.62entropy
6visible tokens
museumtop token

Attention is a content-addressed weighted average.

Score

Each query compares with every permitted key by dot product. Larger alignment produces a larger logit.

sᵢ = q · kᵢ

Scale and mask

Division by √d keeps variance controlled. A causal mask sets future logits to negative infinity before softmax.

z = s / √d + mask

Mix values

Softmax creates nonnegative weights summing to one. The output is their weighted combination of value vectors.

o = Σ softmax(z)ᵢ vᵢ

Temperature changes certainty, not evidence.

Low temperature amplifies logit differences and concentrates probability. High temperature flattens the distribution. It does not create new token relationships; it changes how decisively existing scores are interpreted.

Causal masking enforces information flow.

Decoder self-attention cannot read future positions during training or generation. Masking occurs before softmax so forbidden tokens receive exactly zero probability rather than merely a small weight.

Attention weight is not a complete explanation.

Residual streams, value vectors, multiple heads, MLPs, and later layers determine the final behavior. High weight indicates routing in one head, not necessarily causal importance for the output.

Questions to test

Why divide by the square root of head dimension?

Dot-product variance grows with dimension. Scaling keeps logits in a range where softmax gradients are less likely to saturate.

Does each attention head learn the same pattern?

No. Heads use separate projections and can specialize in local, positional, syntactic, or semantic relations, though specialization is not guaranteed.

Why does entropy fall when temperature decreases?

Lower temperature expands score gaps, concentrating probability into fewer positions and reducing distributional uncertainty.

Primary sources

Vaswani et al. (2017) introduced scaled dot-product multi-head attention. Brown et al. (2020) describes causal transformer language modeling at scale. Jain and Wallace (2019) investigates whether attention weights are explanations. Elhage et al. (2021) develops transformer-circuit interpretations of attention heads.