The Straight Path

The model in Chapter 9 could draw, and it charged fifty forward passes for every picture. The reason was geometric: it had been taught to follow a path that curves, and you cannot take long strides along a curve without leaving it. This chapter asks what happens if you pick a straighter path — and the answer turns out to be shorter, plainer, and, once you have it, steerable in a way the curved version never was.

The last chapter chose carefully between two schedules and never asked whether it needed one of that kind at all. Ask now. Look at what training actually requires: a way to produce a corrupted picture, a number saying how corrupted it is, and a target to be scored against. Any interpolation between a picture and some noise supplies all three. Both of the last chapter's candidates got their shape from making a derivation come out cleanly, and neither was chosen for being convenient to walk along.

So use the shortest one there is. A straight line.

At $\htmlClass{eq-knob}{\tau} = 0$ that is the picture; at $\htmlClass{eq-knob}{\tau} = 1$ it is pure noise; in between it slides evenly from one to the other. Now differentiate it, which takes one line, because a straight line has the same slope everywhere:

That quantity is the velocity: the direction and speed at which this particular picture is turning into this particular noise. Train the network to predict it, with the same squared error as before, and you have swapped one target for another and changed nothing else about the machine — same transformer, same conditioning, same optimizer, same corpus.

Sampling is now the plainest thing in this book. You are standing at some point with an estimate of your velocity, and you want to be a little further along: multiply and subtract. That is Euler's method, from 1768, and it is the entire sampler. No schedule, no square roots, no variance bookkeeping — take the step.

There is one catch, and the plates below will make it look smaller than it is. The straight line is straight for one pair — this picture and this noise. Training sees the pair; sampling never does. Standing at some point halfway out, the model cannot know which picture it is on the way to, so the velocity it reports is an average over every pair that could have put it there — a velocity belonging to none of them. Follow those averages and you trace a curve, not a line.

It is a far gentler curve, and it can be flattened further still by training a second time on the model's own start-and-finish pairs.1 X. Liu et al. · 2023 Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flow ICLR 2023 · arXiv:2209.03003 Three groups posted versions of the idea within a month of each other in the autumn of 2022 — as rectified flow, as flow matching over arbitrary interpolants,2 Lipman et al. · 2023 Flow Matching for Generative Modeling ICLR 2023 · arXiv:2210.02747 and as stochastic interpolants between two densities3 Albergo & Vanden-Eijnden · 2023 Building Normalizing Flows with Stochastic Interpolants ICLR 2023 · arXiv:2209.15571 — and all three are, underneath, the observation that you can regress on a velocity you know in closed form. The lineage runs back to treating a network as the derivative of a trajectory and letting a solver supply the layers.4 Chen et al. · 2018 Neural Ordinary Differential Equations NeurIPS 2018 · arXiv:1806.07366

The claim is testable on this page, so test it. Both models are here: the noise-predicting one from the last chapter and the velocity-predicting one from this, the same size, trained on the same pictures for the same length of time. At every rung of the walk you can ask each of them the same question — given where you are, what finished picture do you think you are heading toward? A straight path should have a confident answer early. A curved one should keep revising.

Now spend a smaller budget. Sampling cost is steps times the cost of one forward pass, and the forward pass is fixed, so the only question to ask of a sampler is how few steps it can get away with. The plate below answers it, and hands over something that was not asked for: at this size, the two models differ more in kind than in stamina.

Be careful about what that proves. Two models this small, given the same handful of hours, cannot settle which parameterization is better in general — and the gap the plate shows is partly the well-known trouble the noise target has when the picture is nearly gone, where a small error in the predicted noise is divided by a very small number on the way back to a predicted picture. The honest summary is that the velocity target was easier to get working here, and that it degrades more gracefully when the budget is cut. Both of those are why production image models are built the way they are. Stable Diffusion 3's two structural choices were a rectified-flow objective — with a new way of choosing which noise levels to train on — and a transformer over patches, in their case one carrying separate weights for the text and image streams.5 Esser et al. · 2024 Scaling Rectified Flow Transformers for High-Resolution Image Synthesis ICML 2024 · arXiv:2403.03206 Strip those refinements and what is left is the pair of choices these two chapters make, at a few thousand times the size. The step count is not a detail of the research; it is most of what a user of one of these systems experiences.

It is also not the end of the line. If a straighter path buys fewer steps, a student trained to take in one step what its teacher takes in two buys fewer again,6 Salimans & Ho · 2022 Progressive Distillation for Fast Sampling of Diffusion Models ICLR 2022 · arXiv:2202.00512 and a model trained to jump from anywhere on the path directly to its endpoint collapses the walk into a single evaluation.7 Y. Song et al. · 2023 Consistency Models ICML 2023 · arXiv:2303.01469 Each of those is the same trade: some fidelity, and a great deal of arithmetic, exchanged for a shorter walk.

Telling it what to draw

Everything you have been shown so far has been unconditional. The model draws a garment — some garment, whichever one the noise it started from happens to become — and you have no say. Getting a say is structurally easy, and the easy version turns out to hide something else.

The model already takes a second input besides the image: the noise level, which enters as a scale and a shift applied inside every block. Widen that input. Alongside the noise level, pass a one-hot saying which of ten garment classes this picture is — boot, bag, shirt. Ten labels, not a sentence. Real systems replace this part with a text encoder trained to put captions and images in one space8 Radford et al. · 2021 Learning Transferable Visual Models From Natural Language Supervision ICML 2021 · arXiv:2103.00020; the wiring downstream of it is unchanged.

Then the trick. During training, throw the label away one time in ten and let the model see the picture with no class attached. One set of weights therefore learns two things at once: what a boot looks like, and what a picture looks like when nothing has been asked for. At sampling time you run both and read the difference.

The middle term is doing something slightly startling. The difference between what the model would draw with the label and what it would draw without it is a vector at every point in image space — it is the direction the class is pulling. Multiplying it by $\htmlClass{eq-knob}{w}$ and adding it back is insisting on that direction harder than the model would on its own. At $\htmlClass{eq-knob}{w} = 1$ you get the conditional model exactly as trained; above that you get a picture more class-like than any real picture, which is why the Stable Diffusion lineage ships this dial with a default up around 7. This model, being small and steered by ten labels rather than sentences, wants much less — its useful setting is nearer two, and the plate below is how you would find that out. This is classifier-free guidance, and it replaced an earlier version that needed a separately trained classifier to supply the push.9 Dhariwal & Nichol · 2021 Diffusion Models Beat GANs on Image Synthesis NeurIPS 2021 · arXiv:2105.05233 Dropping that classifier and getting the same effect from one network's two moods is the whole contribution.10 Ho & Salimans · 2022 Classifier-Free Diffusion Guidance NeurIPS 2021 Workshop on Deep Generative Models · arXiv:2207.12598

Guidance has two failure modes, and the good setting is defined by them rather than by any principle. Turn it down and the label stops mattering. Turn it up and every picture becomes the same over-inked caricature: the model is being pushed toward a region that maximizes class-likeness rather than one that contains pictures, and variety is the first thing it spends.

Between two things

The label is a vector with a single 1 in it. Training only ever showed the model that kind of vector. Nothing stops you, at sampling time, from handing it something else: half a sneaker and half a boot, a shirt that is already becoming a dress. The model was never asked that question, and it answers anyway.

That answer is the evidence that it learned a space of garments rather than ten separate lookup tables. If the weights were a catalogue, the halfway vector would produce garbage, or a hard cut from one class to the other. What it produces is a walk: a sole grows a shaft, a sleeve lengthens into a skirt, through shapes that are in no catalogue.

The size of what is on this page: 2.5 million parameters, trained on 12,000 pictures at 28 × 28, three and a half orders of magnitude below a production image model in parameters and four in pictures. It shows in every soft edge. The recipe is not smaller — the interpolation, the velocity target, the Euler sampler, the dropped condition, the guidance subtraction.

One structural difference remains, and it is where the arithmetic happens. Running diffusion on megapixel images directly is wasteful, so production systems compress first and decode at the end, and the thing doing the compressing is an autoencoder.11 Rombach, Blattmann, Lorenz, Esser & Ommer · 2022 High-Resolution Image Synthesis with Latent Diffusion Models CVPR 2022 · arXiv:2112.10752 The hourglass from Chapter 4 is still in the machine six chapters later, holding the same waist open for the same reason.

And that is the last of them. In the Prologue there was a blind walker on a smooth landscape, feeling the ground for a downhill direction and taking one small step. Everything since has been that walker on richer ground: a curve bent to fit any shape you drew; space folded until two tangled colours came apart; handwriting read out of 784 numbers and no picture; a map that drew itself with nobody holding the pen; a sentence learning to guess its own next word; a boat that found the wind with nothing but a clock; a judge fitted to your eye and then broken by an optimizer; a pocket of numbers taught chess by nothing but games — and, these last two chapters, a machine that is handed nothing at all and hands back a picture. If you want the whole descent laid out, the epilogue lists it, says who to thank, and tells you where to go next.