Attention — Field Foundations

The one-sentence version: Attention is the step where every word looks at every other word and updates its own meaning based on what's relevant — so bank sitting next to river quietly becomes "river-bank." It's how context finally enters the machine, and it's the engine the whole transformer is built around.

Section 0

Why we're here (the problem we're fixing)

End of Module 2, every token had a rich vector — its meaning-location on the map, plus its position in the sentence. But we flagged one last gap: that vector is context-blind.

bank has the same vector whether you mean a river bank or a money bank. It sits at one fixed spot on the map no matter what's around it. A human reading "she sat by the river bank" instantly knows which bank — from the neighbors. The model, so far, can't: its bank vector hasn't looked at river yet.

The fix

Attention lets each word reach out to the others, figure out which ones matter to it, and pull their meaning into itself — so bank next to river ends up meaning something different than bank next to money.

By the end you'll be able to explain

the problem attention solves (context-blindness) · the core "blend-in-the-others" move · Query, Key, Value · scoring relevance (dot product) → attention weights (softmax) · how a word's vector is updated · multi-head attention · why it's the keystone.

Section 1

The core move — every word updates itself by looking at the others

The whole idea in one line, before any machinery:

The core move

A token's new vector = a blend of all the tokens' vectors, weighted by how relevant each one is to it.

Each word gets to gather a little meaning from every other word — taking more from the relevant ones and almost nothing from the irrelevant ones. After the blend, bank is no longer generic; it's bank-having-listened-to-river.

Analogy — the roundtable

Every word in the sentence sits at a round table. To understand itself here, each word turns to the table and asks: "Which of you is relevant to what I mean?" It listens harder to the words that matter, tunes out the ones that don't, and updates itself from what it heard. Every word does this at once.

Concrete example:

"The animal didn't cross the road because it was too tired."

What does it mean — the animal or the road? You know instantly: the animal (roads don't get tired). For the model to know, it must look back at animal and pull in its meaning. That "looking back and pulling in" is attention. Change "tired" to "wide," and it should attend to road instead — same mechanism, different winner.

Section 2

Query, Key, Value — the three things every word makes

How does a word "ask the table a question" and "listen"? Each token turns its vector into three new vectors:

Query (Q)

"What am I looking for?" — the question the word asks the table.

Key (K)

"What am I about?" — a label the word holds up advertising what it offers.

Value (V)

"What I'll hand over if you listen." — the real content the word contributes.

Analogy — a library search desk

Your Query is what you type in the search box. Every book holds up a Key — the label on its spine. The search matches your query against the spines, and from the best matches you actually read the Value — the contents inside. Query = the ask · Key = the advertised label · Value = the real payload.

Where do Q, K, V come from? Each is made by multiplying the token's vector by a learned weight matrix — one for queries (W_Q), keys (W_K), values (W_V).

🔌 Callback to Module 2

Embeddings are only some of the model's weights, with other weights doing other jobs you'll meet later. These Q/K/V matrices are those other weights. They start random and get trained, exactly like the embeddings did. Training is what teaches the model how to turn a word into a good question, a good label, and a good payload.

Section 3

Scoring relevance — Query · Key

Now the word asks its question. It compares its Query against every word's Key (its own included) and gets, for each, a relevance score: "how well does your label answer my question?" The comparison is a dot product — Query · Key.

So for bank, its Query is scored against the Key of river, of the, of sat… and river scores high (training taught bank's "what I seek" to line up with river's "what I'm about"). The filler words score low.

Footnote you'll see in papers: the scores get divided by a fixed number first — "scaled" dot-product attention — purely to keep the numbers from blowing up. The why is unchanged: it's still a relevance score.

Section 4

Softmax — turning scores into a spotlight budget

Raw scores are messy (one might be 14.2, another −3). We need clean attention weights: positive numbers that add up to 1 — splitting 100% of attention across the table. That's softmax.

So bank might spend 80% of its attention on river, a little on itself, and the rest scattered across filler. Those percentages are the attention weights.

80%river

7%bank (itself)

8%sat / by / the …

5%everything else

bank's spotlight budget — almost all of it lands on "river."

Section 5

The update — a weighted blend of the Values

Now the payoff. The word builds its new vector by mixing every word's Value together, in proportion to the attention weights:

new bank = 0.80 × (river's Value) + 0.07 × (bank's Value) + 0.08 × (the/sat's Values) + …

Because 80% of the blend is river's Value, the updated bank vector now leans hard toward the geography meaning. Context-blindness is dead: bank listened to river and changed.

the whole roundtable, for one word:

bank makes its Query ("what kind of bank am I?")
every word holds up its Key (river: "I'm about water/geography")
score = bank's Query · each Key → river scores high, "the" scores low
softmax the scores → attention weights (river 80%, …)
new bank = weighted blend of every word's Value, using those weights
bank now means "river bank."

Run the same five steps in a "money bank" sentence and money wins the spotlight — so the same word comes out meaning something different, purely from its neighbors. And every word does this at once, each building its own updated vector in parallel.

🔌 GPT footnote

In a GPT-style model (the next-token predictor), a word may only attend to the words before it — it can't peek ahead at what it's supposed to predict. Same mechanism, blindfolded to the future.

Section 6

Multi-head attention — several spotlights at once

One round of Query–Key–Value can really chase only one kind of relationship. But language has many at once: which adjective modifies which noun, which pronoun points to which name, subject-to-verb…

The fix is simple: run several attention operations in parallel, each with its own W_Q/W_K/W_V. Each one is a head. Through training, heads specialize:

one head tracks adjacent words,
one tracks pronoun → the noun it refers to (it → animal),
one tracks subject → verb, and so on.

Then their results are combined back into one vector per word.

Analogy — several spotlights

Instead of one spotlight hunting for "relevant words," shine eight at once, each tuned to a different kind of connection — then pool what they all found. That's multi-head attention: many heads, each looking for a different relationship, all at the same time.

Intuition is all you need to read the papers: one attention = one kind of relationship; multi-head = many kinds in parallel.

Section 7

Why this is the keystone

Step back. Every word, in parallel: (1) asks the sentence a question (Query), (2) finds which words are relevant (Key scores → softmax weights), (3) absorbs a blend of their meaning (Values), (4) comes out context-aware.

Stack this operation many times — layer on layer — and the sentence progressively understands itself: early layers catch simple links (adjacency, grammar), later layers catch deep ones (who it refers to, the gist of a paragraph).

“Tokenization chopped the text. Embeddings gave each piece a meaning. Attention is where the pieces finally read each other — where a sentence stops being a row of fixed dots and starts being a thought.”

This is the idea from the 2017 paper that created the transformer — "Attention Is All You Need" (Vaswani et al. [1]). The title is the claim: throw out the older sequence machinery, and attention alone — words attending to words — is enough to build the most powerful language models we have. This module is the engine room of the whole part.

Section 8

The handoff to Module 4

Attention is one piece of the machine, not the whole thing. A real transformer block wraps attention with a few more parts:

the attention layer you just learned (mix in context),
a small feed-forward network (each word "thinks" on its own about what it gathered),
plus residual connections + layer normalization — helpers that keep the stack stable as it grows deep.

Module 4 (The Transformer Block) assembles these into the repeatable unit stacked dozens of times to make a full model. You now hold the hardest piece; M4 is putting the engine in the chassis.

🧠 Recap

The whole module in six lines

After embeddings, each word's vector is context-blind (bank is the same beside river or money). Attention fixes that.
Core move: a word's new vector = a blend of all words' vectors, weighted by relevance — take more from the words that matter.
Each word makes a Query (what I seek), a Key (what I'm about), a Value (what I'll share) — each via a learned weight matrix (the "other weights" from M2).
Relevance = Query · Key (dot product = alignment). Softmax turns scores into attention weights summing to 1 (a spotlight budget).
The word's update = the weighted blend of everyone's Values — bank that spent 80% on river comes out meaning river-bank.
Multi-head = many attention passes in parallel, each catching a different relationship. Stack many layers → the sentence understands itself. The transformer's engine ("Attention Is All You Need," Vaswani et al. 2017).

✅ Now go and use it

Read it, write your own notes, then talk it through

Step 1 — talk it through. This copies a prompt with the whole module already inside it. Paste it into any AI assistant: it asks you questions one at a time, answers the ones you wrote down, and explains why a wrong answer was tempting. It will not test you.

Step 2 — only when you feel ready. The closed-book gate. It carries this module's own questions and tells the assistant to score you strictly and explain what you misunderstood. Write your own notes first — the notes are for encoding, not for shrinking the test, and the questions cover the whole module, not just what you wrote down.

Download the .md

a read b your own notes c discuss d scored check e on to Module 4

Any assistant works — Claude, ChatGPT, Gemini, an open-source model you run yourself. Free tiers are fine. You can dictate your answers instead of typing.

Sources

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser, Illia Polosukhin, Attention Is All You Need, 2017 — arXiv:1706.03762. The paper that introduced the transformer.

← back to the index Free · no account · no tracking