Telling Things Apart

Two chapters ago you fit a curve through points; in the last one you bent space until two tangled classes came apart. Both were the same ritual, and the ritual has a name we never stopped to say: supervised learning. Here is the whole recipe. Collect a dataset of paired examples — questions with their answers already attached:

Choose a family of functions $\htmlClass{eq-out}{\hat{y}} = f(\htmlClass{eq-world}{x};\htmlClass{eq-model}{\theta})$. Choose a loss $\mathcal{L}$ that scores how much a prediction disagrees with the attached answer. Then run the only rule this book uses, $\htmlClass{eq-model}{\theta} \leftarrow \htmlClass{eq-model}{\theta} - \htmlClass{eq-knob}{\eta} \htmlClass{eq-world}{\nabla \mathcal{L}}$, averaged over examples. Fitting a curve was this recipe with x and y both single numbers. The tangled spirals of Chapter 2 were this recipe with x a point on a plane. Nothing in the recipe says x must stay small.

So make x an image. MNIST is the field's old warhorse1 LeCun, Bottou, Bengio & Haffner · 1998 Gradient-Based Learning Applied to Document Recognition Proceedings of the IEEE 86(11) · leon.bottou.org: handwriting collected at the start of the 1990s from Census Bureau clerks and American high-school students, every digit scanned and reduced to a 28 × 28 grid of gray. Read the grid row by row and the picture becomes a list of 784 brightnesses — a single point in $\mathbb{R}^{784}$, exactly as a point on a plane was a list of two coordinates. In that space, “every 7 anyone might write” is not an idea but a place: a sprawling, tangled region, interleaved with the region of 1s and the region of 9s. The classifier's job is Chapter 2's job, word for word — deform the space until the ten regions come apart and flat cuts can separate them. The only thing you lose in 784 dimensions is the ability to watch the bending. From here on, we judge the deformation by its consequences. Before any model touches them, look at the raw material.

Before the training, the shape of the machine. It is the same stack you built in the last two chapters, with two small pieces of machinery bolted on to make ten classes workable. The 784 brightnesses go in one end. The network ends in ten outputs rather than one — a score per digit, a vector called the logits — and those scores are free to be any size, either sign; nothing yet says they are probabilities. So the last step makes them into some: the softmax takes the ten scores and returns ten positive numbers that sum to one. The tallest is the model's answer, and its height is how strongly the model means it.

The squashing step is one line of arithmetic,

— each score exponentiated, then normalized, so the numbers are positive and sum to one. An equation like this is better felt than read, so play with it: a bank of scores drawn from a bell curve, and the belief they become. Watch the biggest score take most — never all — of the probability; widen the spread $\sigma$ and the winner runs away with it; and see how dividing the logits by a temperature before the softmax sharpens or flattens the verdict.

The second piece of machinery is the loss. Ten probabilities came out; one of them belongs to the right answer. That single number is the only one the loss looks at, and the loss is its negative logarithm:

Take the two symbols in turn. $\htmlClass{eq-world}{y}$ is the true label, the answer written on the form by whoever held the pen; it never changes, and it selects exactly one of the ten bars. $\htmlClass{eq-out}{p_{\htmlClass{eq-world}{y}}}$ is the height of that one bar — the probability the model was willing to place on the truth. The nine other bars appear nowhere in the formula, and yet they are punished all the same: they share one unit of belief with $\htmlClass{eq-out}{p_{\htmlClass{eq-world}{y}}}$, so the only way to raise it is to take from them.

The minus log is what turns a probability into a complaint. It is zero when $\htmlClass{eq-out}{p_{\htmlClass{eq-world}{y}}} = 1$ — perfect confidence in the right answer costs nothing — and it climbs slowly at first, then without limit as the probability approaches zero. One part in ten, which is where an untrained model spreading its belief evenly over ten classes starts, costs $-\log 0.1 \approx 2.30$; a tepid but correct 0.6 costs 0.51; a confident 0.99 costs 0.01; and a confident, wrong 0.01 costs 4.61. That asymmetry is the whole design. Cross-entropy is often called the model's surprise at the truth, and being loudly, specifically wrong is the most surprising thing it can do — so that is where the gradient pushes hardest. Every digit in the plate below carries its own surprise, and the loss in the header is simply their average.

One more piece of discipline — the piece that makes this science rather than wishful thinking. A network with a hundred thousand parameters can memorize thousands of examples: loss falling, accuracy climbing, and nothing learned about handwriting at all. Memorizing answers is not learning, so we grade on questions the model has never seen. Of the ten thousand digits below, 8,000 form the training set the gradients flow from, and 2,000 form the test set, locked away from training entirely. The plate reports both scores, and the pair is more honest than either alone: accuracy on rows it has studied, accuracy on rows it has never seen, and the shortfall between them — the memorization it got away with. Only the second number is a claim about handwriting.

Even this discipline leaks. A test set is only held out until people start choosing what to keep by how it scores, and then, one published result at a time, the field begins fitting to it too. When researchers built a fresh test set for a famous image benchmark by repeating the original collection procedure as closely as they could, every model scored worse on the new one — not catastrophically, but consistently, in the order of their original ranks.2 Recht et al. · 2019 Do ImageNet Classifiers Generalize to ImageNet? ICML 2019 · arXiv:1902.10811 Nothing here was cheated. The number was simply optimistic in a way nobody could see from inside.

What the machine actually receives

What the model is given is not what you see. You see a 7 — two strokes, a sharp corner, a thing with a top. It receives a list: 0.00, 0.00, 0.31, 0.89, 0.94, … — 784 numbers in a fixed order, and nothing else. It is not told that pixel 215 sits directly above pixel 243; adjacency, stroke, corner, loop — every visual notion has to be rediscovered statistically, from which brightnesses rise and fall together across eight thousand examples. That “seven-ness” can be recovered from a bare list of numbers is the quiet astonishment of this chapter, and the mechanism is the one you already watched: the same bending of space as Chapter 2, performed in 784 dimensions instead of two.

While it trains, the two curves in the plate tell different stories. The training loss nearly cannot help falling — the optimizer is paid to push it down, and with a hundred thousand parameters there is room to push. The test accuracy is the earned number: performance on questions the gradients never touched. How well a model carries over from the examples it studied to examples it has not seen is called generalization, and the gap between those two curves is exactly where it fails. Here that gap stays small, and the reason is unglamorous: eight thousand examples of a well-posed task. It is the field's bluntest law — more data beats cleverness, almost every time it is allowed to compete. And not vaguely: across vision, speech and language, error falls as a power law in the number of training examples, with enough regularity that you can forecast what another ten times the data will buy before you go and collect it.3 Hestness et al. · 2017 Deep Learning Scaling is Predictable, Empirically arXiv preprint · arXiv:1712.00409

The 2,000 test digits are still other people's handwriting, scanned by the same machines in the same decade. Your handwriting, drawn with a pointer, is a genuinely harder exam. One courtesy is extended before each guess: MNIST digits are centered by their center of mass, so the pad below re-centers your stroke the same way — the model has never seen a digit hiding in a corner, and without that shift it would fail for reasons that have nothing to do with shape.

Beside the pad sits an evidence square, and it arrives with a caution attached. It is the gradient of the winning score with respect to each pixel: brighten this pixel a little, and the score for the model's answer goes up by this much, or down. Laid out as an image, that vector is a saliency map, and it does look like an explanation.4 Simonyan, Vedaldi & Zisserman · 2014 Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Maps ICLR 2014 workshop · arXiv:1312.6034 Read it as one carefully. A gradient describes the model's behaviour in an infinitesimal neighbourhood of exactly this input, which is not the same thing as the reason for the verdict — and some popular relatives of this method have been caught producing the same handsome picture even after the network's weights are replaced with random numbers.5 Adebayo et al. · 2018 Sanity Checks for Saliency Maps NeurIPS 2018 · arXiv:1810.03292 A picture that survives the destruction of the model was never describing the model. The bare gradient shown here is the honest, unretouched version: noisier than the pretty ones, and answering a question it can actually answer.

Opening the box

When it misreads your 4 — and sometimes it will — you can do better than shrug. Nothing about this machine is sealed. The first layer maps 784 pixels onto a bank of hidden units, so each unit owns exactly 784 incoming weights — one per pixel — which means each unit is an image, and you can simply look at it: the pattern of ink it rewards and the pattern it penalizes. Units deeper in are harder to see that way, so there is a second question to ask of any of them — which digits drive you hardest? — and the answer is a picture too, since you can average the winners. And its mistakes on the test set have structure. It does not confuse digits at random; it confuses digits that share a skeleton, the same pairs you would confuse squinting at a bad fax.

The machine's idea of a 7 is further from yours than its mistakes suggest. Take an image it reads correctly and confidently, compute which direction in pixel space would most reduce that confidence, and step a little way along it — a change so small that the two pictures are indistinguishable side by side. The verdict flips, often to a confident wrong answer. These are adversarial examples, and they are not a bug in one model: they were found in every network anyone tried, and the same doctored image usually fools models trained separately on different data.6 Szegedy et al. · 2014 Intriguing properties of neural networks ICLR 2014 · arXiv:1312.6199 The explanation is less exotic than the effect. In a space of 784 dimensions, a nudge too small to see in any single pixel is a large step overall if you take it in the one direction the model is most sensitive to — and a network built out of nearly-linear pieces has such a direction everywhere.7 Goodfellow, Shlens & Szegedy · 2015 Explaining and Harnessing Adversarial Examples ICLR 2015 · arXiv:1412.6572 The model has genuinely learned something about handwriting. It has not learned what you would call seeing.

The workhorse and its price

What you just trained is not a toy version of the real thing; it is the real thing, small. This exact recipe — pixels in, softmax out, cross-entropy pushed downhill — was reading the amounts on several million American bank checks a day by the late 1990s1 LeCun, Bottou, Bengio & Haffner · 1998 Gradient-Based Learning Applied to Document Recognition Proceedings of the IEEE 86(11) · leon.bottou.org, and sorting mail by ZIP code a decade before that.8 LeCun et al. · 1990 Handwritten Digit Recognition with a Back-Propagation Network NeurIPS 1989 · proceedings.neurips.cc Scaled up, it reads street signs for cars and flags tumors on scans. Whenever the world hands you questions paired with answers, supervised learning is the tool you reach for first, and usually the tool that wins.

Its price is hiding in that phrase: paired with answers. Every label in this chapter was a person's judgment, recorded one digit at a time; the recipe consumed eight thousand of them before it could read a single stroke of yours. Labels are supervised learning's fuel and its ceiling — expensive to collect, and the model learns to predict them and nothing more. So delete them. Keep the ten thousand images and burn every answer: is anything left to learn? The next chapter squeezes the digits through a two-number bottleneck with no labels at all — and a map of handwriting appears on its own.