Choosing the Next Word (Logits → Softmax → Sampling)

The one-sentence version: After the last block, the model turns each word's final vector into a score for every word in its vocabulary (logits), softmax turns those scores into a probability wheel, and a sampling strategy spins that wheel to pick the next word — which is exactly why the same prompt can give different answers.

🔌 One wording note

"word" = "token." The model scores and picks tokens, not always whole words. We say "word" because it reads easier, and for common words 1 token = 1 word. Mentally substitute "token" throughout.

Section 0

Why we're here (the last step that makes a word)

Look how far you've come — you hold the whole engine now:

text → tokens (M1) → embedding vectors (M2)
→ attention mixes in context (M3) → stacked blocks refine it on the residual stream (M4)
→ a final, deeply-context-aware vector for each token

After the final block, the last token sits there holding a rich vector — everything the model figured out about "what should come next." But a vector is a list of numbers. The model's actual job was to produce a word.

This module is the bridge from that final vector to an actual chosen token. It's the shortest hop in Part 1, but it answers two things you've probably wondered about:

Why does the same prompt sometimes give a different answer each time?
What is "temperature," the setting you've seen on every AI playground?

By the end you'll be able to explain

what logits are + how the final vector becomes one score per token · what softmax does (scores → a distribution summing to 1) · greedy vs sampling · what temperature does (and why low = safe, high = creative) · what top-k and top-p trim away · why the same prompt can give different answers — and when it won't.

Section 1

From the final vector to logits — the scoreboard

The model has a fixed vocabulary — every token it knows (M1: built by BPE, often ~50,000 for GPT-2, ~100k+ for newer models). To pick the next token, the model has to weigh in on all of them at once.

So it takes the final vector for the current position and produces one number for every token in the vocabulary. That long list of numbers is called the logits.

What a logit is

A logit is a raw, unbounded score — "how much do I like this token as the next one." Higher = more preferred. They are not yet probabilities: they can be negative, can be large, don't add up to anything tidy. Think raw scoreboard, not percentages. A 50,000-token vocabulary → a list of 50,000 scores for this one position.

catlogit 8.2

doglogit 6.1

thelogit 3.0

bananalogit -1.4

xylophonelogit -4.0

the logits = one brightness per word in the whole vocabulary (5 of ~50,000 shown)

📐 2-minute math — where logits come from (the "unembedding")

Remember M2: an embedding matrix turned each token-ID into a vector (one row per token). The output step runs that idea in reverse. The model compares the final vector against every token's vector and gives each a score (basically a dot-product — M3's "how aligned are these two vectors?" — against every vocabulary entry). High score = "this final vector points the same way as that token's vector." The result is the logits. You don't need the matrix mechanics to read papers — just hold: final vector → compared against the whole vocabulary → one raw score per token.

Analogy — the contestant and the giant scoreboard

Picture a game show with 50,000 buzzers, one per word. The model lights up each buzzer with a brightness = how much it wants that word next. cat glows bright, the glows medium, xylophone is nearly dark. That wall of brightness levels is the logits. Now the model has to actually pick one.

Section 2

Softmax — turning scores into a probability wheel

Raw scores are awkward to pick from. You can't "roll the dice" on a list like [8.2, -1.4, 3.0, …] — what's the chance of each? You need them as probabilities: every token gets a clean percentage, and they all add up to 100%.

The function that does this is softmax. Feed it the logits, and out comes a probability distribution — one probability per token, all between 0 and 1, summing to exactly 1.

the probability wheel
slice size = probability

cat ~46% dog ~27% the ~20% banana ~5% … ~2%

Same ranking as the scoreboard — now as slices of one wheel. cat gets a fat slice; xylophone a sliver you'd almost never land on. Pick the next token = spin the wheel.

Two properties worth holding:

It keeps the order. The biggest logit becomes the biggest slice. Softmax never reshuffles who's ahead — it converts the gaps into proportions.
It exaggerates the lead, gently. A token a bit ahead in logits ends up with a more-than-proportional slice. Big leads become near-certainties; small gaps stay genuinely uncertain.

📐 2-minute math — what softmax actually computes

For each logit, two steps: (1) exponentiate it (raise e to that power — e^logit), making every value positive and blowing up the big ones faster; (2) normalize — divide each by the sum of all of them, so the whole set adds to 1. That's it:

probability = e^(this logit) / (sum of e^(every logit))

The exponentiate step is why a leading token gets an outsized slice — e^8 is enormously bigger than e^3. You'll never compute this by hand; just hold: softmax = exponentiate, then normalize → a clean probability wheel that favors the leaders.

Key takeaway

Logits = raw scores (any range, no meaning on their own). Softmax → probabilities (0–1, sum to 1, ready to pick from). Same ranking, now as slices of one wheel.

Section 3

Greedy vs sampling — two ways to pick

You've got the wheel. How do you choose the next token?

Greedy — take the biggest slice

Skip the spin. Pick the single highest-probability token, every time. Deterministic — same wheel → same pick. Safe for short facts, but over a paragraph it gets flat and repetitive (loops, always the most-obvious word).

Sampling — spin the weighted wheel

Pick randomly, weighted by slice size. The fat slice usually wins, but sometimes a medium slice gets picked — and that bit of chance is what makes writing feel alive, varied, human.

Analogy

Greedy = a contestant who always buzzes the brightest buzzer. Reliable, but predictable and boring. Sampling = the contestant spins the weighted wheel — favorite usually wins, but there's room for a surprise. Most real generation uses sampling, then shapes the wheel (next two sections) to control how wild the spin is.

Key contrast — the root of "why different answers?"

Greedy = deterministic (no randomness; biggest slice every time). Sampling = random, weighted (spin; bigger slice = better odds, not a guarantee). Hold that for §6.

Section 4

Temperature — the creativity dial

Here's the famous knob. Temperature reshapes the wheel before you spin it — it decides how sharp or how flat the slices are.

low T (0.2)
sharpened — leader dominates
safe · focused · dull

T = 1.0
the "true" wheel
untouched distribution

high T (1.2)
flattened — long-shots get a chance
creative · risks nonsense

Low temperature (0.2) — sharpen the wheel. Leader's slice fattens, long-shots shrink toward nothing. Nearly predictable. (At the limit, temperature = 0 is exactly greedy — the top slice swallows the wheel.)
High temperature (1.2) — flatten the wheel. Slices even out, unlikely tokens get a real chance. Adventurous — and pushed too far, incoherent.
Temperature = 1.0 — the wheel softmax made, untouched.

Analogy — the dial on the spinner

Low temperature squeezes the wheel so the favorite owns almost all of it (boring but safe). High temperature spreads it so even sliver-tokens get landable slices (creative but risky). You're not changing which token leads — you're changing how much the leader dominates.

📐 1-minute math — how temperature works

Before softmax, you divide every logit by the temperature T. Divide by a small T (like 0.2) → logits get bigger and more spread out → softmax makes a spiky wheel (leader dominates). Divide by a large T (like 1.5) → logits get squished together → flat wheel (everyone's closer). T = 1 divides by 1 = no change. That's the whole mechanism: temperature stretches or squishes the score gaps before they become slices.

Trade-off

Low T = coherent but dull/repetitive. High T = creative but risks nonsense. No universally "right" value — it depends what you want. Code generation likes low T (one right answer); brainstorming likes higher T (variety). Temperature is the coherence ↔ creativity slider.

Section 5

Top-k and top-p — trimming the wheel before you spin

Temperature reshapes the whole wheel, but never fully removes the junk — at high temperature some genuinely terrible token still has a sliver, and once in a while you land on it and derail the sentence. Top-k and top-p fix that by chopping the bad slices off entirely before the spin.

Top-k — keep the k biggest slices

Pick a number k (say 40). Keep only the 40 highest-probability tokens, throw the rest away, rebuild the wheel from just those, spin. The other 49,960 can't be picked at all. Simple: it caps how far down the list a surprise can come from.

Limit of top-k

k is fixed, but how many tokens are "reasonable" changes by context. After "The capital of France is" basically one token is right (Paris) — keeping 40 is too many. After "My favorite hobby is" hundreds are good — 40 is too few. A fixed k can't adapt.

Top-p (nucleus) — keep the smallest set that adds up to p

Instead of a fixed count, pick a probability mass p (say 0.9). Starting from the biggest slice, keep adding tokens until their slices sum to 90% — then stop and discard the rest. Spin from that "nucleus." The clever part: the set resizes itself.

"The capital of France is ___" → model is confident

Paris 95% Lyon 1% Nice 0.5% … everything else

nucleus = basically just Paris (one slice already covers 90%). Adapts to "confident."

"My favorite hobby is ___" → model is wide open

readinghikinggaming cookingpaintingrunning … ~hundreds, until they sum to 90%

nucleus = a big set (no single token dominates). Adapts to "open."

Analogy

Top-k = "keep the top 40 slices, always." Top-p = "keep just enough top slices to cover 90% of the wheel — could be 1 slice, could be 300." Top-p is more popular because it adapts to how sure the model is. Many systems combine top-p and temperature.

Key takeaway

Temperature reshapes the wheel (sharp vs flat). Top-k / top-p trim the wheel (cut the unlikely tail) — top-k by a fixed count, top-p by an adaptive probability mass. Often used together: trim the junk, then shape what's left.

Section 6

Why the same prompt gives different answers

Now you can answer the question the whole module was pointed at.

If decoding uses sampling (spinning the weighted wheel), there's real randomness in every pick. Same prompt → same wheel → but a different spin → a different token → and from there the sentences diverge. That's why an AI can answer the same question two different ways.

And you can now predict exactly when it won't differ:

Greedy / temperature = 0 → deterministic. No spin, always the biggest slice → same answer every time (assuming nothing else changes).
Sampling with temperature > 0 → random → different answers, more varied the higher the temperature.

"A language model doesn't 'know' its answer — it builds a probability wheel for the next token and spins it. Determinism vs creativity is just whether you spin, and how sharp you made the wheel first."

Section 7

The handoff — you've closed the machine

Trace it end to end, because you can now do the whole journey:

"The cat sat on the"
→ tokens (M1)
→ embedding vectors (M2)
→ attention mixes in context (M3)
→ stacked blocks refine on the residual stream (M4)
→ final vector for the last position
→ LOGITS: a raw score for every vocabulary token (§1)
→ SOFTMAX: scores → a probability wheel (§2)
→ SAMPLING: shape it (temperature), trim it (top-k/top-p), spin it (§3–5)
→ next token: "mat"
→ append "mat" to the text, and do the whole thing again for the next token.

That last line is the secret of the whole machine: it produces one token at a time, then feeds the result back in and runs again. A whole paragraph is this loop, over and over.

That is the complete LLM machine — prompt in, word out. Part 1 is done.

🧠 Recap card

The whole module in six lines

After the final block, the model turns each position's final vector into logits — one raw score per vocabulary token (any range, no meaning alone). It's a comparison of the final vector against every token (the "unembedding").
Softmax turns logits into a probability distribution — every token 0–1, all summing to 1. Picture a weighted spinner: slice size = probability. Keeps the ranking, gives leaders an outsized slice (exponentiate, then normalize).
Greedy = always take the biggest slice (deterministic, but flat/repetitive). Sampling = spin the weighted wheel (random, weighted — lively, human). Most generation samples.
Temperature reshapes the wheel before the spin: low = sharp (safe, dull; T=0 is greedy), high = flat (creative, risks nonsense). It divides the logits before softmax — stretching or squishing the gaps.
Top-k keeps the k biggest slices (fixed count); top-p / nucleus keeps the smallest set summing to p (adaptive — 1 token when confident, hundreds when open). They trim the junk tail; often combined with temperature.
Same prompt → different answers because sampling is random. Greedy / T=0 → same answer every time; sampling with T>0 → varied.

✅ Now go and use it

Read it, write your own notes, then talk it through

Step 1 — talk it through. This copies a prompt with the whole module already inside it. Paste it into any AI assistant: it asks you questions one at a time, answers the ones you wrote down, and explains why a wrong answer was tempting. It will not test you.

Step 2 — only when you feel ready. The closed-book gate. It carries this module's own questions and tells the assistant to score you strictly and explain what you misunderstood. Write your own notes first — the notes are for encoding, not for shrinking the test, and the questions cover the whole module, not just what you wrote down.

Download the .md

a read b your own notes c discuss d scored check e Quiz Me on all five

Any assistant works — Claude, ChatGPT, Gemini, an open-source model you run yourself. Free tiers are fine. You can dictate your answers instead of typing.

← back to the index Free · no account · no tracking