Embeddings

Section 0

End of Module 1, your text was a list of token IDs — one integer per brick. But those IDs are arbitrary: cat might be 9246 and dog might be 5011, and the fact that 9246 > 5011 means nothing — cats aren't "more" than dogs. The ID is a coat-check number: it points at the right brick, but it doesn't describe it.

That's a problem, because the model needs to compute with meaning — to feel that cat and dog are related and that cat and democracy aren't. Raw IDs can't give that. Embeddings fix exactly this.

Section 1

Don't worry — this is the only math you need, and it's easy.

Section 2

An embedding is the vector that stands for a token. And getting it is dead simple: a table lookup. The model keeps one giant table — the embedding matrix — with one row per token. Each row is that token's vector:

So the pipeline grows by one step:

The ID's only job was to say which row to grab. Once grabbed, the model holds a list of numbers that describes the token — not a bare name tag. Real models use vectors of a few hundred to a few thousand numbers per token.

Section 3

Because every token is now a point in space, we can talk about tokens being near or far. The model arranges them so that:

Picture a giant map (really hundreds of dimensions, but let's draw it flat):

Section 4

This is the famous one. On the map, it's not just where a word is — the directions between words carry meaning. There's a direction that means "royalty" and one that means "gender." So you can do arithmetic on meaning:

Start at king. Walk the "remove male" direction, then the "add female" direction — and you land right next to queen. Nobody told the model "queen is the female king." It falls out of the geometry, because the map encoded "gender" and "royalty" as consistent directions.

First shown dramatically by a 2013 model called word2vec (Mikolov et al. [1]). Modern LLM embeddings are richer and get refined by context later (that's attention, Module 3) — but this idea is the foundation underneath all of it.

Section 5

Not from a human. The vectors are learned during training.

At the start, every token's vector is random — the map is pure noise, every word scattered nonsensically. Then, as the model trains on trillions of tokens (predicting the next token, over and over), training nudges the vectors (gradient descent). Words appearing in similar contexts get nudged together; words that don't drift apart. After enough training, the random scatter has organized into the meaningful map. Nobody designed the map — frequency and context built it, the same way BPE built the vocabulary in Module 1.

Section 6

A catch we must fix before Module 3. The embedding for dog is the same vector no matter where it appears. But order matters enormously:

If all the model had were the word-vectors, these two sentences would look identical to it. Disaster. The fix: add a second piece of information to each embedding — its position. Token 1 gets a "position-1" signal, token 2 a "position-2" signal, and so on. This is positional encoding. So the real vector that enters the next stage is:

Now dog-in-slot-1 and dog-in-slot-3 are distinguishable, and word order survives. (How the position signal is built has a few flavors you'll meet in papers — but the why is all you need here: order would otherwise be lost.)

Section 7

You now have, for every token, a rich vector saying what the word means and where it sits. A sentence is now a little row of meaningful points in space.

But each token's vector is still context-blind — bank has one vector whether you mean a river bank or a money bank. Module 3 (Attention) fixes that: it lets tokens look at each other and mix their vectors based on context, so bank next to river ends up meaning something different than bank next to money. That's where the sentence starts to understand itself.

🧠 Recap

✅ Now go and use it