Teaching Taste

The learners of chapter 6 were paid by a number the world handed them: a harbour worth twelve, a shoal worth minus nine, a pole still upright. Nobody had to be asked. The reward was a fact about the situation, and the only interesting question was how to divide the credit for it.

Now try to write that number for the things people actually want from a machine. Score this answer for helpfulness. Score this paragraph for tact. Score this ornament for beauty. Each of those is a goal you can recognize in a heartbeat and cannot express as arithmetic, and no verifier anywhere will settle it for you. This chapter is about what you do then. The answer is the most consequential idea in how any assistant you have used was built, and it arrives with a failure mode attached that you are going to inflict on yourself, on purpose, about six minutes from now.

Start with the part everyone gets wrong. The obvious move is to ask people for scores — rate this out of ten — and the obvious move is a disaster. People are miserable absolute raters. Their scale drifts within a single sitting, it drifts further between one person and the next, and a seven from a generous morning means nothing against a seven from a tired afternoon.

But the same person, shown two things at once, is superb. You cannot put a number on a cup of coffee. You know instantly which of two you would rather drink. So do not collect ratings. Collect comparisons: a prompt, a winner, a loser, and no numbers anywhere.

Which needs something to have taste about, so here is a space of printer's rosettes — the sort of engraved device a book like this one would strike on a title page. Each is drawn from six numbers: how many arms, how far they curl, how far they reach, how heavy the stroke, how full the inner bloom, and what ink. Nobody has told the page which regions of that space are any good, and nothing on this page knows until you say so.

From verdicts to a number

Now turn that pile of comparisons into a score, which sounds like alchemy and is in fact ordinary statistics, and older than the name it travels under. Zermelo wrote it down in 1929 to rank chess players from a tournament table1 Zermelo · 1929 Die Berechnung der Turnier-Ergebnisse als ein Maximumproblem der Wahrscheinlichkeitsrechnung Mathematische Zeitschrift 29 · gdz.sub.uni-goettingen.de; Bradley and Terry rediscovered it in 1952 and got their names on it. Posit that every ornament has a hidden worth $\htmlClass{eq-model}{r_\phi}(y)$, and that a person prefers stochastically according to the gap between two worths, squashed into a probability:

That sideways symbol just means was preferred to, and $\htmlClass{eq-op}{\sigma}$ is the same S-curve that has been squashing scores into probabilities since chapter 1. Feel the shape with numbers: a gap of zero gives 0.50, a coin flip; a gap of one gives about 0.73; a gap of three gives 0.95. So the thing being fitted is calibrated in how reliably a person would pick this one, which is a claim you can check.

Then fit it the only way anything is ever fitted here. For each pair, ask how surprised the model was that you picked the winner; average that surprise; walk downhill.

Look at what that is. A model that takes two things, scores each, subtracts, and squashes the difference to predict a winner. That is Elo. That is a credit score. That is every probability-this-deal-closes spreadsheet ever built. The exotic-sounding reward model at the heart of modern alignment is the oldest, most boring shape in statistics, pointed at pairs of essays instead of pairs of chess players — and the mystique evaporates the moment you can see the two forward passes and the subtraction. Bolting a neural network onto that shape and optimising against the result is far more recent than the shape itself: it was first done in 2017, on Atari games and simulated robots, where about a thousand human comparisons bought behaviours nobody could have written a reward function for.2 Christiano et al. · 2017 Deep Reinforcement Learning from Human Preferences NeurIPS 2017 · arXiv:1706.03741

One number below is worth more than the rest. It is easy to score a model on data it was trained on and learn nothing. So before each of your clicks, the judge — knowing only the clicks that came before it — was asked to call the pair in advance. That running score is the only honest thing on the plate.

The proxy and the goal

You now have a number that stands in for your taste, and a number is a thing an optimizer can climb. This is the moment the whole field turns, so slow down for it.

Your judge is not your taste. It is four hundred parameters fitted to a few dozen of your clicks, and away from those clicks it is guessing. Where you never voted, it extrapolates, and extrapolation from a small sample is not opinion but arithmetic accident. There are regions of ornament space where the judge is confidently, splendidly wrong, and neither of you has any idea where they are.

Nor is the industrial version any purer. When this is done for real, the people supplying the comparisons agree with each other only about three times in four,3 Stiennon et al. · 2020 Learning to Summarize from Human Feedback NeurIPS 2020 · arXiv:2009.01325 so the judge is fitted to a noisy majority and inherits that noise as a ceiling — one no amount of extra data will lift.

An optimizer will find them. Not out of malice — it has no model of your intentions and no interest in them. It has a function and a gradient, and the fastest route uphill runs straight through whichever hole is nearest. Goodhart's law: when a measure becomes a target, it ceases to be a good measure. The Soviet nail factory judged by the weight of nails produced a few enormous useless ones; judged by count, a million tiny useless ones. Neither factory was malfunctioning. Both were optimizing exactly what they were told. (That factory is a cartoon from a Soviet satirical magazine rather than a documented plant, which does not make it less exact.)

The plate below turns an optimizer loose on your judge with no restraint of any kind, and photographs the policy as it travels — the distance measured in nats of KL from where it started, which is the field's honest ruler for how hard did we optimize. Two curves get drawn on the same axis. One is your judge's opinion, which costs nothing and can be computed a million times a second. The other is yours, and every point of it costs you a click. Watch where they part company, and notice that nothing on the first curve marks the spot.

This is not a quirk of a four-hundred-parameter judge on a toy space. Run the same experiment with real reward models and real language models and you get the same two curves, in the same arrangement, reliably enough to fit an equation to: the proxy climbs steadily with distance travelled while the true score rises, peaks and turns over — and where it turns over moves predictably with how large the reward model is and how much data it was fitted to.4 Gao, Schulman & Hilton · 2023 Scaling Laws for Reward Model Overoptimization ICML 2023 · arXiv:2210.10760 The failure has a shape, and the shape has parameters. That is a good deal better than knowing it can happen, and still nowhere near being able to see the peak from the inside.

The leash

So the optimizer must be stopped, and the question is what to charge it for. Not distance in the parameters — parameters lie about behaviour, and always have. Charge it for how far its behaviour has drifted from something already trusted. Distance between two distributions has a standard meter, and here at last is the debt chapter 5 left unpaid:

Read the subscript before anything else, because the subscript is the safety property. The average is taken over draws from your own policy, so the penalty only ever notices places you actually go — and it bites hardest exactly where you are putting mass the reference never would, which is the definition of the blot. Written the other way round it would patrol territory you never visit and shrug at the exploit. KL is not symmetric, and here the asymmetry is not a wart to apologize for. It is the entire mechanism.

Put the two halves together and you have the single most important objective in modern AI:

This is the objective the assistant you have talked to was actually trained on5 Ouyang et al. · 2022 Training Language Models to Follow Instructions with Human Feedback NeurIPS 2022 · arXiv:2203.02155. In English: score as high as you can while remaining recognizable. $\htmlClass{eq-knob}{\beta}$ is the length of the leash — the exchange rate between reward points and strangeness. Push it to infinity and the policy never moves. Push it to zero and you get the previous plate. Everything worth having is in between.

And this objective does not need a search. It has an exact answer, which you can write down in one line and which is the same shape you will meet every time a paper says soft, temperature, or log-sum-exp:

Take the distribution you already trust and tilt it exponentially by reward. Boost, don't replace. Feel why it has to be this before checking that it is: you want mass where reward is high, and you are charged for leaving home, so the answer must be home multiplied by something that grows with reward. The plate draws all three curves — the reference, the tilt, and their product — along whichever gene you like, and hands you β.

One more fence, for a different fear

The leash prices drift. There is a second instrument for a problem that sounds similar and is not, and it belongs to the fact chapter 6 kept circling: a policy generates its own training data. In ordinary supervised learning a step that is too large is an inconvenience — the dataset sits there, patient and fixed, and the next few steps recover. Here, a destructively large step collapses the policy into some degenerate habit, and every sample it collects from then on is garbage, so the gradients computed from that garbage cannot guide you back. You did not stumble. You sawed off the branch you were standing on.

So the question becomes: what is the biggest step that is still safe? PPO's answer6 Schulman et al. · 2017 Proximal Policy Optimization Algorithms arXiv preprint · arXiv:1707.06347 is disarmingly cheap. Weight each action by the ratio between what the new policy thinks of it and what the policy that collected the data thought — $\rho = \htmlClass{eq-model}{\pi_\theta} / \htmlClass{eq-mute}{\pi_{\theta_{\text{old}}}}$ — and then compute both the honest objective and a fenced one, and always take whichever is worse for you.

Don't memorize it — walk it. The design is the insight, and it is asymmetric on purpose.

The heist

One last thing, because it is the prettiest result in this part of the field and it needs no plate — only the two equations already on this page, read in the other direction.

Look again at the exact solution, $\htmlClass{eq-model}{\pi^{*}} \propto \htmlClass{eq-mute}{\pi_{\text{ref}}} \exp(\htmlClass{eq-model}{r}/\htmlClass{eq-knob}{\beta})$, and instead of solving for the policy, solve for the reward:

Sit with what that says. Any policy that is optimal for some leashed objective is carrying its own reward function around inside it, and that reward is nothing but the log-ratio of the policy to the reference. Ranking things by how much more likely fine-tuning made this than the reference would have is ranking them by reward. Your language model is secretly a reward model.

That normalizer $Z$ is a sum over every string a keyboard could produce, which is less a large number than a category error, and for years it made the formula a pretty footnote. Then somebody noticed where the reward was going to be used: inside $\htmlClass{eq-op}{\sigma}(\htmlClass{eq-model}{r}(\htmlClass{eq-world}{y_w}) - \htmlClass{eq-model}{r}(\htmlClass{eq-world}{y_l}))$, which touches rewards only through a difference between two responses to the same prompt. Same prompt, same $Z$, and the uncomputable term annihilates itself. What is left is one supervised loss on preference pairs that provably targets the same optimum — no reward model to train, no sampling, no rollouts, no critic. That is DPO7 Rafailov et al. · 2023 Direct Preference Optimization: Your Language Model is Secretly a Reward Model NeurIPS 2023 · arXiv:2305.18290, and the whole four-model circus collapses into ordinary fine-tuning.

With one bill, which this book will not hide. DPO never generates anything, so it can never discover a response better than the ones already in its dataset; all it can do is re-weight what is already there. To get reliably better than your data, the model has to produce, be judged, and update. That gap has been measured, and it falls where you would expect: widest when the good responses are rare in the dataset, which is the case anyone actually cares about.8 Tajwar et al. · 2024 Preference Fine-Tuning of LLMs Should Leverage Suboptimal, On-Policy Data ICML 2024 · arXiv:2404.14367 There is no way around the loop.

What you just did to yourself

Step back and count the moves, because every one of them has an industrial twin. You gave comparisons, not scores, because comparisons are the only judgment people give reliably. A Bradley–Terry head turned that pile into a number. An optimizer aimed at the number found the places the number was wrong, and your own eye — expensive, slow, and the only thing that was ever the goal — peaked and turned over while the dashboard kept climbing. A KL leash back to something trusted was what stood between the two. Swap ornaments for essays and four hundred parameters for a trillion, and that is RLHF, unchanged in every particular.

It also tells you exactly where this pipeline is weakest, and therefore where the next chapter goes. Everything here rests on a judge that had to be learned, from a small and expensive pile of human clicks, and could therefore be wrong in ways nobody can see. Suppose instead the judge were a program that cannot be flattered, cannot be bribed, and is never wrong — a referee that simply knows. Then there is no proxy to game, no annotator's taste in a box, and no peak to fall off.

That judge exists for some questions and not others, and which side a question falls on does more than anything else to decide whether machines are getting rapidly better at it. Meet Rook — and watch plate VI turn out to be machinery it barely needs.