Every Number Has a Parent

This blog has been a blank for three months.

The last post went up on the 19th of July. Then nothing.

Not because there was nothing to say. Because I went back to being a student, one subject at a time.

First, Inference Engineering, Philip Kiely’s book on serving models in production.

Then the early-access chapters of Boni García’s Context Engineering.

Then the GPU Programming Specialization from Johns Hopkins on Coursera, which asks you to think in threads and blocks and memory a writer never has to care about. Until now.

Three subjects, in that order. Serving a model fast. Feeding a model the right context. The silicon underneath both.

The two books open from the same unspoken sentence, and the course is about the hardware that sentence runs on.

Assume you already have a trained model.

Fine. Where did it come from?

Most of us can recite the answer. Weights. Gradients. Backpropagation. Very few of us could point at a single weight and say who put it there, what it started as, or what moved it. When a model card says 70 billion parameters, what is one parameter, physically? A number. Fine. Whose number?

So today this blog starts a series. I’m calling it Weights to Words: first how the weights get made, then how those weights turn into the words you read.

• Part 1, this post: how a model gets trained. Where every single number comes from.
• Part 2, next, and onward: it picks up exactly where this post ends. What happens after this kind of training. The fine-tuning and human feedback that turn a raw text predictor into an assistant. Then the model leaves school, and the hard engineering moves to serving it, feeding it context, and the GPUs doing the arithmetic. That is the territory those three months of reading covered.

And this one isn’t a diagram post. It comes with a real model.

A tiny one. Forty-four trainable numbers, one attention head, three possible answers. It reads “I drink hot” and has to guess the next word. The 44 break down like this: 6 numbers for the word embeddings, 12 in the attention machines, 17 in a small feed-forward block, and 9 in the answer judges. You’ll meet every one of them. Nothing here is a cartoon of learning. It genuinely makes a guess, genuinely works out which of its numbers to blame for the mistake, and genuinely adjusts them with Adam, the optimizer family the big labs still build on. Each of those steps gets its own lab below, and you train it with a button in Lab 07.

Here it is right now, before anyone has taught it anything.

It picked coffee. Fifty-eight percent sure. The book says tea.

Why coffee? Because when we set the starting numbers by hand, the coffee judge got a head-start bonus of +0.60 and the tea judge got −0.35. Nothing learned that. We typed it. You’ll meet those judges in Lab 05.

Nothing is broken. That is simply what 44 hand-set numbers believe about hot drinks.

Every number on this page belongs to one of three families, and the colors never lie about which:

• Chosen. A person put it there and nothing will change it. The sentence. The token IDs. The learning rate.
• Trainable. A person set its starting value, and training is allowed to turn it. These are the parameters. The weights. The knobs.
• Calculated. Made fresh on every run, by a fixed rule, from numbers that came before.

Every number in a language model was chosen by a person, set as a knob and turned by training, or calculated by a rule. There is no fourth kind.

That sentence is the whole post. The eight labs below are the proof, one stage at a time. The box above shows how to use them: step through the tabs, tap any colored number, open the drawers when you want the arithmetic.

Two ways to read this. Straight through, skipping the labs: the text stands on its own. Or stop at every lab and poke the numbers. Either way, this model is simplified on purpose, one attention head, one layer, three words, so the bottom of every stage has a note called “What a real LLM does instead.” Nothing here pretends to be production scale.

The name tags

Before the model learns anything, a surprising amount has already been decided.

The sentence is chosen. The IDs are chosen: hot is 43 because we said so, and 43 means nothing, the way your employee ID says nothing about your work. Then the ID fetches a row of two numbers from the embedding table, and here is the first real twist. Those two numbers are already knobs. They are not “the meaning of hot.” They are a starting guess that training will shove around until it becomes useful. In a real model those starting values are small random numbers from a seeded generator. Here they are typed in, so the page behaves the same for everyone.

The position stickers in stage 4 matter more than they look. Attention, on its own, treats a sentence like a bag of words. Without the sticker, nothing tells the model which word came first. In this toy the sticker is simply added. Most modern models encode position a different way, a trick called RoPE that works inside the attention step you’re about to meet. Same job.

Three machines with four screws each

This is where most explainers lose people. Slow down here.

Q, K and V. Query, key, value. The names sound like a database. They are not. The query is what a word is looking for. The key is what each word offers to be matched against. The value is what a word hands over if it gets picked. Each is made by its own little machine with its own four knobs, and the same machine runs on every word in the sentence.

The distinction worth keeping is the one between the machine and what it makes. Wq is stored. It is part of the model. Q, the pair of numbers it produces for “hot”, is thrown away after every run and rebuilt from scratch next time.

If you write docs for a living, you already know this shape. Wq is the template. Q is the rendered page. You maintain the template. The page gets regenerated on every build.

The knobs are the model. Everything else is a calculation the knobs make possible.

The room where words listen

Here is the famous part. The one with the paper title everybody quotes.

“Hot” holds its Q card up against every word’s K card, including its own. Each comparison is a multiply-and-add, divided by √2, the square root of the card length, so the scores don’t blow up as cards get longer. Softmax, a fixed recipe (raise e to each score, divide by the total), turns the three scores into three shares of one pie. Those shares decide how much of each word’s V message gets mixed into hot. Then the original hot is added back on top, so the original isn’t thrown away.

That’s attention. A weighted average, with weights the model computes for itself.

Look closely at the softmax stage. The untrained model listens to all three words almost equally: 33.5, 34.0 and 32.5 percent.

Remember those numbers. They come back.

The second workshop

One more machine, and it is the least glamorous.

The pair from attention goes into a feed-forward block: three little calculators widen it from two numbers to three, a fixed bend called tanh squashes each one, and a second table folds the three back down to two. The bend is the important part. Without it, stacking layers of multiplication collapses into one big multiplication, and depth buys you nothing.

Unglamorous, and enormous. In a standard transformer layer, the feed-forward block holds roughly two thirds of the parameters.

Three judges

Here is the strangest fact in the whole pipeline.

The judges never see the word “hot.” They never see “drink.” By this point, everything the model understood about the sentence has been compressed into one final pair of numbers, and that pair is all the judges get. Each judge, one per possible answer, has its own two knobs and a bonus, and turns the pair into a raw score called a logit. Softmax, the same recipe from attention, turns the scores into chances.

In this toy there are three judges. Llama 3 has a vocabulary of about 128,000 tokens. That’s 128,000 judges, every single time it produces a word.

Up to here, the model has only guessed. Nobody has graded it.

The mistake meter

The answer card finally comes out of the drawer.

One honest word about “wrong” first. Coffee isn’t bad English. I drink hot coffee is a perfectly good sentence. In training, the target is simply whatever the text actually said next, and this page of text said tea. Across a whole library, both words would earn probability, roughly in proportion to how often each one follows. Wrong here means one thing only: not what this example said.

The book says tea. The model gave tea 22.7 percent. A fixed formula, the negative log of that chance, turns it into one number: 1.4815. That is the loss. If the model had given tea 100 percent, the loss would be zero. The smaller the chance, the steeper it climbs. That’s the reason for the log: it punishes confident misses hard. A 50 percent chance costs 0.69. A 1 percent chance costs 4.6.

Then the clever part. Backpropagation walks the whole calculation backward, from the loss through the judges, the workshop, attention, Q, K, V, all the way to the embedding rows, and hands every one of the 44 knobs a gradient: if you turn this knob up a hair, here is how much the mistake changes. It works because each step only needs to know one local fact, how much its output moves when its input moves, and a rule from school calculus, the chain rule, multiplies those local facts together on the way back.

One gradient you can check on a napkin. For the tea bonus, it is just the chance of tea minus one, a clean shortcut you get from pairing softmax with this log loss. 0.227 minus 1. That’s −0.773.

Most of them you can’t. So the lab proves one the hard way. It takes a single Q knob, nudges it up by 0.0001 and reruns the entire network, nudges it down and reruns again, and measures the slope directly. Backprop says 0.000194411. The wiggle says 0.000194411. They agree to the twelfth decimal place.

Play with the sandbox in stage 20, the “Prove it” tab. Pick the tea bonus and the curve is steep: that knob matters a lot right now. Pick the Q knob and the curve is nearly flat. Turn it as far as you like. The mistake barely notices.

That’s what a gradient really is. Not a mysterious signal. A measurement, for this one example and these exact knob settings, of how much each knob would move the mistake, and in which direction. Change the example or the settings and the measurement changes too.

The coach

Knowing which way to turn a knob is not the same as turning it.

That’s the optimizer’s job, and this toy uses Adam, from Kingma and Ba’s 2014 paper. It keeps two running memories for every knob, the average gradient and the average squared gradient, and divides one by the square root of the other. Dividing a gradient by its own typical size cancels the size out. What’s left is mostly direction.

Here’s the one nobody expects. Before training, the tea bonus has a gradient of −0.77. One of the K knobs, Wk[1][0], has a gradient of −0.000146. A shout and a whisper, more than five thousand times apart. Ten rounds later, that whispering K knob has moved 0.43, one of the biggest moves on the whole board. Adam cares far less about how loud a gradient is than plain gradient descent does. What moves a knob far is a gradient that keeps pointing the same way.

One press of its button is one round: guess, grade the guess, backprop, nudge all 44 knobs.

Now press it.

One round: tea goes from 22.7 to 29.0 percent. Two rounds: 36.9. Five: 73.2. Ten: 99.4. Coffee collapses to almost nothing.

Then scroll back up the page. Every lab has recalculated. The embedding dots in Lab 01 have drifted. The Q, K and V cards have new values. The judges’ knobs have turned. Same sentence, same rules, same arrows. Only the knobs changed, and that changed everything downstream.

That is the core loop of pretraining, the first and by far the biggest phase of training a language model. Real pretraining runs this same loop with more than just more data. It predicts the next token at every position of every sentence at once, in batches of millions of tokens, through dozens of layers, with a mask so no word can peek at the words after it. Bigger in every direction. Same loop.

What ten rounds actually taught it

Ninety-nine percent sure it’s tea. Impressive.

Now go back to Lab 03 and look at the attention shares again. After ten rounds they read 33.4, 35.1 and 31.5 percent. Barely moved.

It didn’t learn to listen to “drink.” It didn’t learn that “hot” goes with beverages. It learned to say tea. Every knob got nudged until the right answer fell out for the one sentence it has ever seen. It is a student who memorized the answer key and never opened the textbook, and the training loss cannot tell the difference. That is exactly why real labs grade their models on text the model never trained on.

Don’t take my word for it. Test it.

After ten rounds, “I I I” gets tea at 97 percent. So does “hot drink I”. Before training, every one of them got coffee. The model never read the sentence at all.

This is why real training needs an absurd amount of text. GPT-3 had 175 billion parameters and trained on about 300 billion tokens. Llama 3 trained on over 15 trillion. Big models still memorize some of what they see: Carlini and colleagues found memorization grows with model size and with how often a passage is duplicated in the training data. But no model can memorize its way through fifteen trillion tokens. The cheapest way to push the loss down across all of it is to learn patterns that generalize: grammar, facts, the shape of an argument.

Same loop. Same three kinds of numbers. Just enough sentences that cheating stops working.

Here’s why I think this matters for anyone who writes documentation.

The question this page keeps asking, where did this number come from, is the question we ask for a living. Where did this claim come from? Which version? Who verified it? A trained model is the first artifact most of us will document where the honest answer is “from fifteen trillion tokens and an enormous number of tiny turns of a knob.” You don’t need to write CUDA to document AI well. But if you can’t trace even a toy model end to end, you are not documenting the system. You are repeating its marketing.

Closing the answer book

At some point a human decides training is done. Usually that means the planned budget of text and compute has been spent, or the score on text the model has never seen has stopped improving. They save the knobs and close the answer book for good.

What’s left is inference: text in, tokens out, no loss, no gradients, no updates. Just the forward pass, again and again, one word at a time.

Look at the loop in that last lab. Inference runs steps 1 to 8 and stops.

Everything in those three months of reading lives inside those eight steps. Context engineering decides what text goes in at step 1. The K and V cards from Lab 02 get saved instead of recomputed for every new token: that is the KV cache, and it is a large part of Inference Engineering. Quantization stores the knobs in fewer bits. The GPU course is about making the multiplications in steps 4 through 7 run on thousands of cores at once.

That is Part 2. The model has left school. Now someone has to run it fast and cheaply, for millions of people, with the right context in front of it.

The reading list

If this lab made you want more, these are the primary sources, not summaries of summaries.

• micrograd, Andrej Karpathy’s tiny autograd engine. The backprop in this toy follows the same pattern: every number remembers what made it.
• Let’s build GPT: from scratch, in code, spelled out, Karpathy again, building a real character-level transformer in two hours.
• Attention Is All You Need, the 2017 paper that introduced the transformer.
• Adam: A Method for Stochastic Optimization, the optimizer in Lab 07.
• The Illustrated Transformer, Jay Alammar’s visual walkthrough, still the best picture book for this.
• 3Blue1Brown’s neural network series, for the intuition behind gradients and backprop.
• And the three that kept this blog quiet: Inference Engineering (free to read online), Context Engineering, and the GPU Programming Specialization.

The blank

Scroll back to the top of this page.

The sentence is still there. I drink hot, and a blank.

If you trained the model, the blank now says tea. Nobody typed it. Forty-four numbers moved a little, a few dozen times, and the blank filled itself in. If you didn’t train it, it still says coffee, struck through in red, because of a bonus we typed by hand.

This blog had a blank too. Three months of it, July to October, spent reading about everything that happens after a model is trained. Part 2 is the after. This was the before: every number, and every parent.

Before you go, make it guess right.

Press train.