Skip to content

How the Transformer works

The Transformer is a neural architecture that processes a sequence through attention. Each position can combine information from other positions without traversing them one by one as a recurrent network does. That property made training highly parallelizable and enabled models for text, images, audio and video to scale.1

Attention is the distinctive operation, but a complete Transformer also needs embeddings, positional information, projections, feed-forward networks, residual connections and normalization.

The architecture in one view

ARCHITECTURE
A Transformer block, end to end
Attention mixes information across positions; the feed-forward network transforms each position. Residual paths preserve a direct route.
Attentionmoves information across positions
FFNtransforms information within each position
Residualpreserves a direct path for signal and gradients

In an autoregressive model, the final representation is projected onto the vocabulary to produce next-token probabilities.

1. Embeddings and positional information

A token begins as an integer identifier. A learned matrix turns it into a vector. Tokens that occur in similar contexts can end up with related representations.

Attention alone does not know order. The model must add or otherwise incorporate position. The original paper used sinusoidal positional encodings added to the input embedding. Later architectures use learned positional embeddings, relative positions or rotational transformations such as RoPE.12

TRANSFORMER INPUT
Token identity + position: order changes the relationship
Self-attention does not encode order by itself. The model needs a positional signal to distinguish sequences with nearly the same tokens but different relationships.
SEQUENCE A
modele(model)position 1 · PE(1)
→
correctede(corrected)position 2 · PE(2)
→
evaluatore(evaluator)position 3 · PE(3)
who acts: the model → who receives: the evaluator
SEQUENCE B
evaluatore(evaluator)position 1 · PE(1)
→
correctede(corrected)position 2 · PE(2)
→
modele(model)position 3 · PE(3)
who acts: the evaluator → who receives: the model
Original Transformer xi = e(tokeni) + PE(i) The embedding and positional encoding have the same dimension and are added before entering the stack.
Later families Position does not have to be added to the embedding RoPE, for example, introduces position through rotations applied to Q and K inside attention.
Identity ≠ orderThe token embedding represents which unit it is; the positional signal indicates where it appears.
Same tokens ≠ same sequenceChanging positions can reverse relationships even when the vocabulary is nearly identical.
There is no single mechanismSinusoidal addition describes the original Transformer; learned, relative and RoPE positions use different mechanisms.
Primary sources: Vaswani et al. (2017) · Su et al. (2021, RoPE).

Position is not cosmetic: it lets the model distinguish who acts on whom even when a sequence contains nearly the same tokens. The exact mechanism is not universal; embedding + position describes the original Transformer, while RoPE incorporates position through rotations inside the attention computation.2

2. Query, Key and Value

Each representation is projected into three vectors:

  • Query (Q): what information this position is looking for.
  • Key (K): what signal each position offers to be matched.
  • Value (V): what content it contributes if it receives attention.

Scaled attention connects four operations: projecting Q, K and V; computing compatibility; normalizing the weights; and mixing the values.1

SELF-ATTENTION
The attention equation, turned into a flow
Q encodes what each position is looking for; K determines compatibility; V provides the content that is ultimately mixed.
1Project
X
→
Qqueries
Kkeys
Vvalues

Three learned projections of the same representation.

2Compare
S = QKT / √dk

Each row compares one query with every key.

3Normalize
A = softmax(S)

Each row is normalized so its weights sum to 1.

4Mix values
Z = AV
α₁v₁+α₂v₂+…→z

The output is a weighted combination of the values.

Attention(Q, K, V) = softmax(QKT / √dk) V
The colors are a teaching aid, not weights from a specific model. The real operation runs on matrices.

The product QKᵀ calculates compatibility between positions. The √d_k factor controls logit scale. softmax turns each row into normalized weights. Multiplication by V produces a weighted combination of information.

The explanation “every word looks at all the others” is useful, though incomplete. Each head learns different projections and can specialize in different patterns.

3. Multi-head attention

Instead of running one attention operation over the full dimension, the block divides the representation into several heads. Each head computes its own Q, K and V matrices.

MULTI-HEAD ATTENTION
Multiple projections read different relationships before recombining
Each head receives the same sequence, learns its own Q/K/V projections and produces a contextual output. The head outputs are then concatenated and an output projection WO mixes them again.
EACH HEADheadi = Attention(QWiQ, KWiK, VWiV)
OUTPUTMultiHead = Concat(head₁, …, headₕ) WO
SHAREDthe same input sequence
NOT SHAREDthe Q/K/V projections
NO FIXED ROLESa head has no universally preassigned semantic role
The matrix patterns are illustrative. The architecture allows different attention subspaces; it does not imply that a particular head always has a stable, interpretable function.
Primary source: Vaswani et al., Attention Is All You Need (2017), §3.2.2.

One head may capture local dependencies. Another may relate distant entities. Another may help copy structure or track delimiters. There is no fixed universal assignment, but the separation increases the capacity to represent multiple relationships simultaneously.

4. Causal masking in generative models

An autoregressive decoder must not see the future during training. A triangular mask prevents position t from attending to later tokens.1

AUTOREGRESSIVE DECODER
The causal mask separates parallel computation from access to the future
During training, many positions are computed at once, but each query can attend only to its own prefix.
keys →12345
queries ↓12345
✓×××× ✓✓××× ✓✓✓×× ✓✓✓✓× ✓✓✓✓✓
visibleblocked: future token
TRAINING
Parallel across rows

Known positions in the batch are computed together. The mask invalidates logits that point to the future before softmax is applied.

q₁q₂q₃q₄q₅
GENERATION
Sequential for new tokens

The next token does not exist yet. The KV cache reuses prefix keys and values instead of recomputing them at every step.

t₁→t₂→t₃→t₄ ?
Contract: position t may use 1…t; never t+1…n.

So although all known positions in a batch can be processed in parallel during training, every prediction follows the same contract that will exist during generation: it can only use the available prefix.

During inference, generation is still sequential because the next token does not exist until the previous one has been selected. The KV cache avoids recomputing keys and values for the whole prefix at each step.

5. The feed-forward network

After attention, each position passes through a dense network independently, using the same parameters for every position in that layer.1

FEED-FORWARD NETWORK · PER POSITION
Attention mixes information across tokens; the FFN transforms each position independently
After attention, each vector passes through the same transformation in the layer: expansion to an internal dimension, a nonlinearity or gate, and projection back to dmodel. No lateral mixing between positions happens during this operation.
ATTENTIONmixes across positions
A position can incorporate information from other positions.
FFNindependent transformation
Each position is transformed independently, but shares that FFN's parameters within the layer.
GENERAL FORM FFN(x) = W₂ σ(W₁x + b₁) + b₂ Original Transformer: ReLU between two projections; dmodel=512 and dff=2048. Modern architectures use different widths and may use gated variants such as SwiGLU.
ATTENTIONdecides which positions to gather information from
FFNlocally transforms the resulting representation
PARAMETERSare shared across positions in one layer; they differ across layers
Primary sources: Vaswani et al., Attention Is All You Need (2017), §3.3 · Shazeer, GLU Variants Improve Transformer (2020).

In large models, this part contains a significant fraction of the parameters and compute. Modern architectures use activations and gates such as GELU, SwiGLU or related variants. Mixture-of-experts models replace one dense network with multiple experts and route each token to a subset of them.

6. Residuals and normalization

Each sub-block does not simply replace the representation it receives: it learns a transformation on top of a residual path that preserves the input. Normalization placement determines where the signal is modified before it continues through the stack of layers.

RESIDUAL + NORMALIZATION
Normalization placement changes the path followed by the representation
Each sublayer learns a transformation F —attention or FFN— while a residual path preserves the input representation. Post-norm and pre-norm place normalization at different points in that same circuit.
POST-NORM · ORIGINAL TRANSFORMER y = Norm(x + F(x))
The identity path also passes through normalization before reaching the next layer.
PRE-NORM y = x + F(Norm(x))
The identity remains a direct path across layers; normalization lives inside the branch that learns the correction.
REPEATEDonce around attention and once around the FFN
UNCHANGEDthe residual idea: the sublayer learns a correction to x
CHANGESwhere the signal passing through the block is normalized
The 2017 Transformer used post-norm. Xiong et al. analyzed why LayerNorm placement affects optimization stability and found better-behaved gradients at initialization for pre-norm in their analysis and experiments; that does not imply one variant is universally superior for every architecture.
Primary sources: Vaswani et al., Attention Is All You Need (2017), §3.1 · Xiong et al., On Layer Normalization in the Transformer Architecture (2020).

The original Transformer used post-norm: it first adds the sublayer output to the residual path and normalizes afterwards.1 In pre-norm, normalization is applied inside the sublayer branch, leaving a direct identity path across layers. Xiong et al. analyze why that placement changes gradient behavior at initialization and can improve optimization stability.3

The same residual structure surrounds both attention and the feed-forward network. The exact normalization detail varies across model families, but the central idea remains: each sublayer learns a correction to a representation that can also advance through the residual path.

Encoder, decoder and encoder-decoder

TRANSFORMER FAMILIES
Encoder, decoder and encoder-decoder: the attention contract changes
They share Transformer blocks, but they do not expose the same information to each position. The difference is what can attend to what and how information flows from input to output.
ENCODERRepresent the full input
ABCD
bidirectional self-attention→contextualized representations

Each position can combine information from the entire input. Q, K and V come from the same sequence.

DECODERGenerate without looking into the future
x₁x₂x₃x₄
causal self-attention→p(xt+1 | x≤t)

Each position can only use the available prefix. The same contract underpins autoregressive generation.

ENCODER-DECODERRepresent the input and generate the output
ENCODER
ABCD
input memory · K,V
Q → cross-attention ← K,V
DECODER
y₁y₂y₃
causal self-attention

Queries come from the decoder; keys and values come from the encoder. This lets the output consult the entire input while remaining causal over its own tokens.

EncoderFull visibility within the input.
DecoderCausal visibility within the output.
Encoder-decoderCausality in the output + cross-attention over the full input.
Key idea: “Transformer” does not describe one single diagram. It describes reusable blocks whose attention pattern changes with the task.
Primary sources: Vaswani et al. (2017), §3.2.3 · Devlin et al. (2018, BERT).

BERT popularized bidirectional encoder stacks for representations useful in classification, extraction and language understanding.4 GPT-style families popularized the causal decoder for generation. The original Transformer combined both: the encoder represented the full input and the decoder generated the output while consulting those representations through cross-attention.1

“Transformer” therefore does not imply one single diagram. It describes a family of blocks and attention contracts.

Why it displaced recurrent networks

RNNs and LSTMs update a state step by step. That creates a sequential dependency that is difficult to parallelize and forces distant information to travel through many steps.

RECURRENCE → SEQUENTIAL DEPENDENCY · ATTENTION → DIRECT INTERACTION
Why attention changed the path between positions
The difference is not that an RNN ‘cannot remember’ while a Transformer can. It is the geometry of computation: a recurrent network propagates state step by step; self-attention can connect two known positions within one layer and process those positions in parallel.
PARALLELISMduring training, the known positions in an attention layer can be computed together; recurrence introduces a step-by-step dependency
SHORT PATHa distant relationship does not need to traverse a chain of recurrent states within the layer
TRADEOFFdense attention compares pairs of positions: it gains parallelism and a short path, but introduces the n × n matrix explained in the next section
IMPORTANTThis describes computation within a layer over positions that are already known. An autoregressive decoder still generates new tokens sequentially: token t+1 does not exist until t has been chosen.
Primary sources: Vaswani et al. (2017), Attention Is All You Need, Table 1 · Sutskever et al. (2014), Sequence to Sequence Learning with Neural Networks

The advantage applies over positions already known during training: attention reduces the number of sequential operations within a layer and shortens the path between distant positions. The price is dense pairwise interaction, which matters increasingly as context grows.

The cost of attention

For a sequence of length n, the QKᵀ matrix contains n × n elements. Its memory footprint and part of its compute grow quadratically with sequence length.

That does not mean total model cost is always O(n²). Projections and feed-forward networks also matter, and optimized implementations avoid materializing some intermediates. But the growth of all-to-all interaction remains a structural limit for very long contexts.

DENSE ATTENTION COST
Sequence length grows linearly; pairwise interactions grow quadratically
In dense self-attention, each of the n queries is compared with n keys. The structural bottleneck appears in the n × n score matrix, not in a claim that the entire Transformer always costs O(n²).
n = 416 scores

4 queries × 4 keys = 16 compatibility scores.

2× tokens→4× scores
n = 864 scores

8 queries × 8 keys = 64 compatibility scores.

What grows with n
ALL-TO-ALL INTERACTIONattention scores · ~ n²

For fixed per-head dimensions, the work of comparing every position with every other position grows quadratically with sequence length.

OTHER BLOCKSprojections + FFN · ~ n

With fixed model width, these operations are applied per position. That is why total model cost is not correctly summarized as “everything is O(n²).”

EXACT KERNELFlashAttention · less memory traffic

It uses tiling to avoid materializing the full attention matrix in HBM and to reduce reads/writes. It still computes exact dense attention.

As n grows, there are several points in the design space
CHANGE THE PATTERNlocal or sparse attentionreduces which pairs interact
REDUCE WHAT ENTERSretrieval, compression, external memoryavoids attending to irrelevant context
CHANGE THE ARCHITECTURESSMs and hybridsexplore a different sequence dynamic
n × n IS THE KEY OBJECTthe score matrix represents pairwise interactions between positions
OPTIMIZING MEMORY ≠ CHANGING THE PATTERNFlashAttention improves data movement without turning exact dense attention into linear attention
O(n²) DOES NOT DESCRIBE THE WHOLE MODELFFNs, projections, and other components also contribute to total cost
Primary sources: Vaswani et al. (2017), Transformer · Dao et al. (2022), FlashAttention · Gu & Dao (2023), Mamba

Research directions include:

  • local or sparse attention;
  • compression and external memory;
  • retrieval of relevant chunks;
  • more efficient kernels;
  • state-space models;
  • hybrid architectures.

Mamba showed that selective state-space models can process sequences with linear scaling and remain competitive across several domains.6 That does not make the Transformer obsolete. It opens another point in the design space.

Transformers beyond text

The architecture operates on sequences of vectors, not exclusively on words.

Vision Transformer divides an image into patches, projects each patch to a vector and applies a Transformer encoder.5 Audio systems can use frames or acoustic tokens. Video systems combine spatial and temporal structure. Multimodal models can align text, image and audio in shared spaces or connect them through cross-attention.

The Multimodality in Generative AI series develops those design choices.

What the architecture does not explain by itself

Knowing the Transformer is not enough to explain a model's behaviour. Other important factors include:

  • pretraining data;
  • the loss objective;
  • the tokenizer;
  • scale and compute budget;
  • instruction tuning;
  • preference optimization;
  • context and tools during inference.

Two models with similar blocks can behave very differently because of the rest of the system.

Where to go deeper in 5sigmas

Frequently asked questions

Is attention the same thing as memory?

No. Attention combines representations available in the current context. A KV cache preserves keys and values for reuse during generation, but it is not persistent memory and does not guarantee remembering information between sessions.

Why divide by the square root of the dimension?

As key dimensionality grows, dot products tend to have larger variance. Dividing by √d_k prevents overly extreme logits and keeps softmax in a more useful gradient regime.

Do all Transformers generate text?

No. An encoder can produce representations or classifications. A Vision Transformer can classify images. Autoregressive generation is one configuration, not a required property.

Does a larger context always improve the result?

No. It increases available information, but also cost and the difficulty of locating relevant evidence. Quality depends on position, noise, long-context training and retrieval strategy.

Primary sources

  1. Ashish Vaswani et al., Attention Is All You Need, 2017. ↩↩↩↩↩↩↩

  2. Jianlin Su et al., RoFormer: Enhanced Transformer with Rotary Position Embedding, 2021. ↩↩

  3. Ruibin Xiong et al., On Layer Normalization in the Transformer Architecture, 2020. ↩

  4. Jacob Devlin et al., BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding, 2018. ↩

  5. Alexey Dosovitskiy et al., An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale, 2020. ↩

  6. Albert Gu and Tri Dao, Mamba: Linear-Time Sequence Modeling with Selective State Spaces, 2023. ↩