The Mechanism

The whole thing, uncompressed

A decoder-only transformer — the architecture behind essentially every current language model — is a function from a sequence of integers to a matrix of scores. That is the entire contract. Everything else is detail about how the function is computed.

Here is the whole pass, with nothing hidden. Four dimensions carry the argument throughout: B is batch size, T is sequence length in tokens, C is the model dimension (the width of the residual stream, called hidden_size or d_model in configs), and V is vocabulary size.

# 0. OUTSIDE THE NETWORK — tokenization
"The cat sat"  →  [791, 8415, 7731]              # (T,)  ints, not floats

# 1. EMBEDDING — integers become vectors
x = W_embed[ids]                                  # (B,T,C)

# 2. N IDENTICAL BLOCKS — each one reads the stream and adds to it
for block in blocks:
    x = x + attention(norm(x))                    # (B,T,C) — unchanged shape
    x = x + mlp(norm(x))                          # (B,T,C) — unchanged shape

# 3. FINAL NORM
x = norm(x)                                       # (B,T,C)

# 4. UNEMBEDDING — vectors become scores over the vocabulary
logits = x @ W_unembed                            # (B,T,V)

Read that again and notice the thing that is easy to miss: the shape never changes in the middle. From step 1 to step 3, the tensor is (B,T,C) the entire way. Forty blocks, eighty blocks, a hundred and twenty — the object being passed along is the same size at every point. That is not an implementation convenience. It is the central architectural fact, and Module 3 is about what follows from it.

The step that is not a neural network

Step 0 deserves separate billing because it is not learned by gradient descent, it happens before the model sees anything, and it is responsible for a startling share of confusing model behaviour.

Tokenization is a deterministic, greedy compression scheme — usually byte-pair encoding — fitted once on a corpus and then frozen. It merges frequent byte sequences into single symbols. Common English words become one token. Rare words fragment. Numbers fragment in ways that depend on the exact digits. Whitespace attaches to the following word in some schemes and not others.

The consequences are not cosmetic:

• Arithmetic is harder than it looks because the digits of a number may not align with token boundaries in any consistent way, so "carry the one" is not a local operation in the model's input representation.
• Character-level tasks are structurally awkward. Counting the letters in a word the model sees as a single opaque symbol requires it to have memorised the spelling of that symbol rather than to look.
• Cost and context length are measured in tokens, not characters, and the tokens-per-character ratio varies by more than a factor of two across languages, which is a real and under-discussed pricing asymmetry.

Byte-pair encoding as a compression algorithm is due to Gage (1994); its application to neural text was Sennrich, Haddow and Birch (2016), who introduced it to handle rare and unseen words in machine translation. If a model's behaviour on some input seems inexplicable, tokenization is the first place to look and it is right an unreasonable share of the time.

Start from something that already makes sense

A Python dictionary does a hard lookup. You supply a key, it finds the one entry whose key matches exactly, and it returns that entry's value. One winner, no partial credit, and — crucially for our purposes — no derivative. You cannot ask a dictionary how its output would change if the key were slightly different, because the answer is a step function.

Attention is that operation made differentiable, in four moves.

1. Every position emits three vectors instead of being a key-value pair. The query is what this position is looking for. The key is the label this position advertises. The value is what it will hand over if selected. All three are linear projections of the position's current residual-stream vector, so all three are learned.
2. Replace exact matching with a similarity score. Dot the query at position i against the key at every position j. High dot product means "these match."
3. Replace "pick the winner" with a softmax. The scores become weights that are positive and sum to one. Now every position contributes something; the good matches just contribute more.
4. Return a weighted average of the values rather than one value.

That is the whole operation. Hard lookup becomes soft lookup, and soft lookup has a gradient everywhere — which means the model can learn what to look for and what to advertise, by backpropagation, from nothing but next-token prediction.

Why divide by the square root of the head dimension

This is the detail people skip, and it is the difference between a model that trains and one that does not.

Take two vectors of dimension d whose components are independent with mean zero and unit variance. Their dot product is a sum of d such products, so its variance is proportional to d and its typical magnitude grows like the square root of d. At a head dimension of 128, raw scores routinely land in the tens.

Now feed scores of that size into a softmax. Softmax exponentiates, so a gap of ten between the largest and second-largest score becomes a ratio of about 22,000 to one. The distribution collapses to nearly one-hot — and the gradient of a saturated softmax with respect to its inputs is nearly zero. The layer stops learning. Dividing by the square root of the head dimension rescales the scores back to order one, which is where softmax has usable gradients.

The causal mask

A language model must not see the future. When predicting the token at position t+1 from positions 0 through t, position t must not be allowed to attend to t+1 — otherwise the answer is in the input.

The implementation is one line: before the softmax, set every score where j > i to negative infinity. Exponentiating negative infinity gives exactly zero, so those positions receive zero weight and, importantly, receive zero gradient. Masking after the softmax would be a bug: the masked entries would still have consumed probability mass from the normalisation.

Why it beat recurrence — the honest answer

The usual story is that RNNs forget long-range context and attention does not. That is real, and it is secondary. The primary reason attention won is that it parallelises and recurrence does not.

An RNN computes the state at position t from the state at position t−1. That is a sequential dependency by construction: step 500 cannot start until step 499 finishes. A GPU with tens of thousands of arithmetic units can therefore do almost nothing at once along the sequence axis and spends the run mostly idle.

In attention, every position's output depends on the full input sequence but on no other position's output. So all positions compute simultaneously, as one large matrix multiply — precisely the operation the hardware is built for. The paper is titled Attention Is All You Need; a fair subtitle would have been and it parallelises.

Hold on to this, because it generalises. Architecture in this field is jointly determined by mathematics and by silicon, and when a design choice looks arbitrary the hardware is usually the missing half of the explanation. That framing is what makes the training and inference stages legible rather than a list of tricks.

The picture in your head is probably wrong

Almost every diagram draws a transformer as a stack: input enters at the bottom, block 1 transforms it, block 2 transforms that, and so on to the top. It is the natural way to draw a deep network and it will actively mislead you.

Because of the residual connections, a block does not transform its input. It reads its input, computes something, and adds the result back:

# NOT this — a pipeline of transformations
x = block(x)

# but this — read, compute, add back
x = x + attention(norm(x))
x = x + mlp(norm(x))

So there is a single vector per position — the residual stream — running unbroken from the embedding to the final norm. Every sublayer is a side channel that reads from the stream and writes an increment into it. Nothing is ever replaced. Things are only ever added.

The term and the framing come from Elhage et al., A Mathematical Framework for Transformer Circuits (Anthropic, 2021), which treats the stream as a communication channel that components read from and write to. It is the single most useful reframing available for this architecture, and it costs nothing to adopt.

Four things this explains — and what each claim actually supports

1. Deleting a middle layer degrades gracefully

Delete stage 7 of a seven-stage pipeline and everything downstream receives garbage. Delete one addend from a sum of 81 and the sum is slightly wrong. The additive structure predicts graceful rather than catastrophic degradation, and that is what is observed — the effect was first characterised for residual networks generally by Veit, Wilber and Belongie (2016), who argued residual networks behave like ensembles of relatively shallow paths.

The limit, stated honestly: this is not uniform across depth. Work on layer pruning in LLMs — for example Gromov et al., The Unreasonable Ineffectiveness of the Deeper Layers — finds that a substantial fraction of the deeper layers can be removed with modest degradation, while early layers cannot. So the correct claim is "later blocks are individually low-impact," not "any block is deletable." If you repeat the strong version you will be wrong in front of someone who has tried it.

2. Activation steering is writing in the model's native format

If a concept corresponds to a direction in the residual stream, then adding a multiple of that vector at some layer is exactly the same kind of operation the model's own sublayers perform. You are not injecting a foreign signal; you are doing what block 12 does, with a vector you chose. The additive view is why this is even a coherent thing to attempt.

The limit: "concepts are directions" is a working hypothesis with substantial supporting evidence, not a theorem. Steering often works, sometimes fails, and can degrade unrelated capabilities. Treat it as an empirical technique with a good mechanistic story rather than a settled account of representation.

3. Depth buys refinement, not re-representation

Layer 30 does not receive a foreign encoding produced by layer 29. It receives the accumulated draft that all 29 previous sublayers have been editing, in the same basis it started in. The model is doing successive editing, not successive translation. This is why intermediate layers can be decoded with the output unembedding at all — the "logit lens" technique — which would be meaningless if each layer re-encoded into a private space.

4. Why low-rank adaptation is plausible

If capability changes correspond to modest edits in a shared space rather than to wholesale rewiring, then a small low-rank perturbation to each weight matrix is a sensible thing to expect to work. That is not a proof that LoRA works, but it is why the idea is not obviously absurd, and it is the intuition to carry into Stage 3.

The division of labour

Say it as one sentence and keep it: attention moves information between positions; the MLP transforms information at a position. Once you hold that, several empirical results stop being surprising — that factual recall interventions tend to localise to MLP layers, that mixture-of-experts routes MLPs rather than attention heads, and that long-context work is almost entirely attention work.

The MLP itself is deliberately boring: project up to a wider dimension (d_ff, called intermediate_size in configs), apply a nonlinearity, project back down. The width ratio is conventionally about four in the classic form. Whatever computation happens at a position happens in that wide middle.

Now count the parameters, because the count is where the intuition lives

This is arithmetic you should be able to do on a whiteboard. Per block, ignoring biases and norm scales, which are negligible:

That is where the familiar one-third / two-thirds rule comes from, and it is exactly right for GPT-2. Check it against a published number: GPT-2 small has C = 768, 12 layers, d_ff = 3072, vocab 50,257, learned position embeddings for 1,024 positions, tied embeddings.

If your hand-count lands on the published number, you have understood where the parameters are. That is a better test of this module than any quiz.

What "knowledge lives in the MLP" does and does not mean

You will hear that the MLP is where facts are stored, often justified by the observation that it looks like a key-value memory: the up-projection produces a sparse-ish activation pattern, the down-projection sums the corresponding rows. Interventions that edit specific factual associations do typically target MLP weights in the middle layers, and they work well enough to be a research subfield.

Be careful with the strong reading. What is supported is that editing MLP weights at particular layers reliably changes particular factual outputs. What is not supported is that each fact has a tidy address, or that MLPs are a lookup table with attention as the index. Model editing has known failure modes — edits that do not generalise to paraphrases, edits that damage unrelated behaviour, edits that revert under further training. Hold the localisation result; do not upgrade it to a filing cabinet.

Pre-norm versus post-norm

The 2017 paper puts the normalisation after the residual addition: x = norm(x + sublayer(x)). Every current model puts it inside the branch: x = x + sublayer(norm(x)). That reordering looks like a detail and is not.

In the post-norm arrangement there is no unnormalised path from input to output — every residual addition is immediately renormalised, so gradients are repeatedly rescaled on the way back. Xiong et al. (ICML 2020) showed that post-norm transformers have large expected gradients near the output layer at initialisation, which is why the original recipe needed a learning-rate warm-up to train at all, and that pre-norm transformers are well behaved at initialisation and can drop the warm-up entirely.

Pre-norm gives you the clean picture from Module 3: an unbroken additive stream, with each branch seeing a normalised copy. GPT-2 made the switch and effectively nobody has switched back.

RMSNorm, and what it dropped

LayerNorm does two things per position: subtract the mean, then divide by the standard deviation, then apply a learned scale and shift. RMSNorm (Zhang and Sennrich, NeurIPS 2019) keeps only the rescaling: divide by the root-mean-square of the vector, apply a learned scale. No mean subtraction, no bias.

The finding that made it stick is that the mean-centring was not doing much of the work. Dropping it removes a reduction pass over the vector and a subtraction, which matters because normalisation is a memory-bandwidth-bound elementwise operation executed twice per block — exactly the kind of cheap-looking op that shows up embarrassingly high in a profile. Same quality, fewer bytes moved. Llama and essentially every model that followed use RMSNorm.

Positions: the model has no idea what order the tokens are in

Attention is permutation-equivariant. Shuffle the input positions and the outputs shuffle with them — the mechanism has no notion of sequence at all. Order has to be injected deliberately, and how you inject it determines what happens when you exceed the lengths seen in training.

RoPE (Su et al., 2021) is worth understanding properly because it is what you will meet. Split each query and key vector into two-dimensional pairs, and rotate each pair by an angle equal to the position times a frequency that depends on which pair it is. Low-index pairs rotate fast, high-index pairs rotate slowly — so the vector encodes position at multiple scales at once, like the hands of a clock.

The property that makes it work: the dot product of two rotated vectors depends only on the difference of the two angles. So attention scores become a function of relative position automatically, without ever adding a position vector to the residual stream. Position information is injected into the query-key comparison only, and never occupies stream bandwidth.

One attention pattern is not enough

A single attention operation produces one softmax distribution per position: one set of weights, one blended answer. But a token frequently needs several unrelated things at once. Resolving a pronoun means finding its antecedent. Getting the verb agreement right means finding the subject. Continuing a list means finding the list's previous items. One distribution cannot point at three places with full weight.

So run several attentions in parallel over slices of the stream. Split the C-dimensional vector into H heads of dimension C/H, run the whole query-key-value-softmax operation independently in each, concatenate the results, and pass them through one output projection. Total compute is roughly unchanged — you split the width rather than adding to it — and you buy H independent routing patterns per layer.

Grouped-query attention is a memory decision

In classic multi-head attention every head has its own W_K and W_V, so at inference you must cache one key vector and one value vector per head per token. That cache is what Stage 5 will show you is usually the binding constraint on how many users a GPU can serve at once.

Grouped-query attention keeps all H query heads but gives them fewer key-value heads to share. Llama 3 70B has 64 query heads and 8 key-value heads, so each key-value head serves eight queries. The cache shrinks by exactly that ratio — eight times — and so do the K and V parameter matrices, which is half of what shifted the parameter split in Module 4.

The quality cost is small and the memory saving is enormous, which is why it is universal. But notice what kind of decision this is: an architecture choice made to reduce inference memory, accepted at a small quality cost, before the model was ever trained. That is the mathematics-and-silicon pattern from Module 2 showing up again, and once you see it you will see it everywhere in modern configs.

What heads "do", and where that story stops

You have read that particular heads perform particular functions. Some of that is genuinely established. Induction heads — which find a previous occurrence of the current token and copy what followed it — are a real, mechanistically characterised circuit, described in Olsson et al. (2022), and their formation is visible as a discrete phase change during training. Previous-token heads are similarly well characterised. These are not stories; they are circuits people have traced.

The whole objective

Show the model a prefix. Ask for a distribution over the next token. Penalise it by the negative logarithm of the probability it assigned to the token that actually came next. Average over every position of every document. That is the entire pretraining objective.

There is no auxiliary term for reasoning. No supervision signal for truthfulness. No objective encoding helpfulness or harmlessness. Everything that later looks like capability is a side effect of compressing a very large corpus under that single loss.

It is worth being concrete about the units, because they let you sanity-check a training run. Natural-log cross-entropy is in nats per token. Divide by ln 2 to get bits per token — the number of yes/no questions you would need to identify the next token given a perfect code. Exponentiate the loss to get perplexity, which behaves like an effective branching factor: perplexity 20 means the model is about as uncertain as if it were choosing uniformly among 20 options.

Why this is what makes post-training necessary

Take the objective literally and the behaviour of a base model stops being mysterious. Ask a faithful next-token predictor "What is the capital of France?" and it may well continue with "What is the capital of Germany?" — because in a very large corpus, that string most often appears in a list of quiz questions. The model is not confused. It is doing exactly what it was optimised to do.

A base model is a document completer. It has enormous knowledge and no notion that it is supposed to answer you. Post-training — Stage 3 of this path — exists entirely to close that gap, and it is much easier to understand once you have accepted that pretraining was never trying to produce an assistant.

This also gives you the sharpest available refutation of a common enterprise mistake. If the loss is next-token prediction over your documents, then fine-tuning on your documents optimises the model to produce text like your documents. It does not optimise for knowing what is in them. That is why fine-tuning does not reliably add facts, and the argument is available to you from the objective alone, before any empirical evidence.

The fields that matter

A config.json is a small document and most of it is boilerplate. Nine fields carry the architecture:

The three derivations to have at your fingertips

Run the third one on Llama 3 70B: 2 × 80 × 8 × 128 × 2 bytes = 327,680 bytes, or 320 KiB, per token. At the advertised 131,072-token context that is 41 GiB for a single sequence — more than half an H100's 80 GB, on top of 141 GB of weights that already do not fit on one card. Without grouped-query attention it would be eight times that.

What the config cannot tell you

Be equally clear about the limits, because config-reading breeds overconfidence. The file gives you shape and therefore cost. It says nothing about the training data, the number of tokens trained, the post-training recipe, or quality at any context length short of the maximum. Two models with byte-identical configs can differ by an enormous margin on every benchmark. Architecture determines what a model costs; data and training determine what it knows.

The tool asks you to guess the MLP's share of the parameters before it computes one. That ordering is deliberate: if you see the answer first you will find reasons to have expected it, and the useful case is the one where your guess and the arithmetic disagree. Module 4 gave you two worked examples that point in different directions, so this is a fair test of whether you took the right lesson from them.