The Transformer Block
& Residual Stream

Section 0

Module 3 gave you the hardest piece: attention, where every word looks at the others and pulls in context. By the end of one attention step, bank had listened to river and updated itself.

But attention is one operation. A real model — GPT-2, GPT-4, Claude — isn't one attention step. It's dozens of layers stacked on top of each other, run one after another. Two questions fall out of that, and this module answers both:

1. What exactly is the repeatable unit that gets stacked? (The transformer block — attention is only part of it.)
2. What carries the information from one layer up to the next, so 96 layers can build on each other without losing the thread? (The residual stream — the most-referenced object in every interp paper you'll read.)

Section 1

A transformer block has two processing steps, and they do opposite kinds of work. This contrast is the thing to lock in.

What does the FFN actually do? Intuition is enough: it's a small network that takes the word's context-filled vector, expands it, runs a non-linear step (so it can represent more complex things), and shrinks it back. This is widely believed to be where a lot of the model's stored factual knowledge and per-word computation lives. You don't need the matrix math to read papers — just hold: FFN = each token's private thinking step, right after attention's group discussion.

Section 2

Here's the crucial design choice, and it's simpler than it sounds. When attention or the FFN produces an output, the block does not throw away the word's old vector and replace it. It adds its output on top of what was already there:

That running, ever-accumulating vector — the thing that flows up through the whole stack, getting added to at every step — is the residual stream.

Think of it as a notebook riding the conveyor belt. Each worker doesn't rewrite the notebook from scratch — they read what's in it, write their addition in the margin, and pass it on. The notebook keeps everything: the original word, plus every layer's contribution stacked up.

Why add instead of replace? Two big reasons, both worth holding:

1. Nothing gets lost. If layer 3 needs the original word identity, it's still right there in the stream — earlier information stays available to every later layer. Replacing would erase the past at each step.
2. The deep stack stays trainable. Adding creates a clean, uninterrupted path for the learning signal to travel back down during training (backprop). Without these add-back paths, a 96-layer model's signal would fade to nothing on the way down and the bottom layers would never learn.

Section 3

There's one more part in a block, with a humble job: keep the numbers sane.

When you stack dozens of layers, each adding to the residual stream, the values can drift — some blow up huge, some shrink tiny. That makes training unstable (the model thrashes or stalls). Layer normalization (LayerNorm) fixes this: right before each worker does its job, it re-centers and re-scales the vector so its numbers sit in a clean, consistent range. Then the worker operates on the tidied-up version.

Footnote you'll meet in papers: there's a debate about putting LayerNorm before each sub-layer ("pre-norm") vs after it ("post-norm"). Modern models mostly use pre-norm because it trains more stably. Same role either way — don't get snagged on it.

Section 4

Now put the pieces together. One transformer block, start to finish, operating on the residual stream:

That unit is one block. Notice the rhythm: normalize → mix (attention) → add back → normalize → digest (FFN) → add back. Gather, then think — writing both contributions into the same running notebook.

Section 5

A single block is the unit. The model is that block, repeated — its own fresh weights each time:

Each block reads the residual stream the block below it produced, does its gather-then-think, and writes its refinement back. So the stream gets progressively richer as it climbs:

• early layers catch simple, local things — word identity, grammar, which word sits next to which;
• middle layers assemble those into phrases, references, simple facts;
• late layers hold abstract, whole-sentence meaning — who it refers to, the sentiment, the gist, what word comes next.

Section 6

Here's the payoff that makes this module load-bearing if you go on to read alignment research.

Because every block reads from and writes to the same residual stream, that stream is a single, continuous record of everything the model is computing — at every layer, for every token. A researcher can tap the stream at any layer and inspect what's encoded there, or edit it and watch how the output changes.

That's exactly what these phrases (which you'll see constantly) mean:

• "we read off the residual stream at layer 8" → we looked at the carried-along vector at that depth to see what the model had figured out so far;
• "we patched the residual stream" → we swapped in a different vector at some layer to test what that information was causing (activation patching);
• "the feature is written into the residual stream" → some concept (e.g. "this text is in French") shows up as a direction in that carried vector.

Section 7

You now hold the whole engine: text → tokens (M1) → embedding vectors (M2) → attention mixes context (M3) → stacked blocks refine it on the residual stream (M4). After the final block, each word has a final, deeply context-aware vector.

But the model's job was to produce a word — the next token. So one question remains: how does that final vector become an actual chosen word?

Module 5 (Output: Logits → Softmax → Sampling) closes the machine: the final vector → a score for every word in the vocabulary (logits) → softmax turns those into probabilities → a sampling strategy picks the next token (and is why the same prompt can give different answers). After M5 you'll have traced one token's entire journey, prompt in to word out.

🧠 Recap

✅ Now go and use it