Make sense of
long AI answers.

Imagine reading a long answer and reaching the word ‘it’. To understand that word, you often need to look back: does it refer to the experiment, the result, or the question? The useful context might be nearby, or several sentences away. A model needs some way to connect the word it is processing with information elsewhere in the text. Attention provides a way to make those connections without treating every surrounding word as equally useful.

The weights form a mixture of the value vectors:

For example, weights of $0.7$, $0.2$, and $0.1$ give $z=0.7v_1+0.2v_2+0.1v_3$.

Some earlier sequence models compressed an input into a single fixed-size representation before producing an output. That created a bottleneck, especially for longer inputs. Attention instead lets a position draw on multiple available representations, assigning different weights to them. The key idea is selective access: keep the information available, then decide which parts contribute to the current representation. These weights are learned computations, not a person consciously choosing where to look.

Attention starts by projecting each token’s representation into three vectors: a query, a key, and a value. The query describes what information this position is looking for. The key supplies something other positions can compare their queries against. The value contains information that can be carried into the result. These are numerical vectors produced by learned transformations, not labels or questions written in ordinary language.

A library analogy helps separate the roles. Think of the query as a search request, the keys as catalogue entries, and the values as the contents of the books. Matching a request against catalogue entries tells you which contents may be useful. Attention differs from a typical library search because it usually mixes information from multiple positions instead of returning just one item. The same token can supply a query, a key, and a value, because it can both seek context and provide context to other tokens.

Each query is compared with every allowed key by taking a dot product. This produces a score for each possible connection. The scores are divided by the square root of the key dimension, then a softmax converts them into nonnegative weights that sum to one. Finally, those weights are applied to the corresponding value vectors, and the weighted values are added together. The result is a new representation containing a mixture of information from the positions that were available.

The complete calculation is:

Here $d_k$ is the key dimension. With $d_k=64$, divide the scores by $\sqrt{64}=8$ before applying softmax.

Why divide before applying softmax? As the vector dimension grows, unscaled dot products can become large in magnitude. Very separated scores can make softmax concentrate almost all its weight on one position, leaving weak gradients for learning. Scaling helps control that effect. Separately, a mask can exclude connections that must not be used. In causal language modeling, for example, a token cannot attend to future tokens. The weighting happens only over the positions the mask allows.

One attention operation gives each position one way to combine context. Multi-head attention performs several such operations using different learned projections. Each head works with its own queries, keys, and values, so the heads can learn different patterns of relationships. Their outputs are combined and projected back into a representation that the rest of the model can use. This gives the model several routes for moving information between positions.

It is tempting to describe one head as handling grammar and another as tracking meaning. That can be a useful intuition, but it is not a job assignment built into the architecture. The roles emerge during training and are not necessarily neat or consistent. Attention is also only one part of a transformer block. Other components, including feed-forward transformations and residual connections, help process and preserve the information. Stacking blocks allows later layers to operate on representations already shaped by earlier context.

Word order changes meaning. ‘The dog chased the person’ and ‘The person chased the dog’ contain the same words but describe different events. Attention needs access to information about position to distinguish sequence order reliably. Comparing content vectors alone does not give the attention operation a complete account of where each token sits. Transformer architectures therefore include mechanisms that make position or relative position available to the computation.

There are several ways to do this. Some architectures add position representations to token representations. Others modify the attention computation so that relationships between positions influence the scores. The practical point is the same: content and order both matter. Position information does not give a model unlimited memory or guarantee that it will use every detail correctly. It gives the model a signal it can learn to use when deciding how information from different parts of the sequence should interact.

In standard full attention, each position is compared with every position in the sequence. With n positions, that creates an n-by-n collection of scores for each head. Doubling the sequence length therefore creates four times as many pairwise scores, not twice as many. This quadratic growth helps explain why handling very long inputs is a computational challenge, even when the individual attention calculation is straightforward.

Efficient implementations can avoid storing the entire score matrix at once, and alternative attention patterns can reduce which positions interact. These approaches have different tradeoffs, so the simple quadratic description does not capture every implementation’s memory use. To remember the basic mechanism, keep the sequence of operations in mind: form queries, keys, and values; compare allowed positions; scale and normalize the scores; then mix the values. That repeated process lets a transformer build representations in which each token’s meaning is informed by its context.