Rook

Every large language model you have talked to was made in three acts. Pretraining: predict the next token on an enormous, indiscriminate corpus, until the model speaks the language of its data. Supervised fine-tuning: keep training, but on a small corpus somebody curated, until it speaks the way you want. And a last act of reinforcement learning: let the model generate, let a judge score what comes back, and move the weights toward whatever scored well. The whole modern pipeline fits in one sentence — pretrain on everything, fine-tune on the good stuff, reinforce on verified outcomes.

This chapter runs all three acts on one small transformer, live, in this page. Not a diagram of the pipeline — the pipeline, with every gradient computed on your GPU. The model is called Rook, and it learns chess.

Chess, first, because moves are tokens. A move written in UCI notation — e2e4, g8f6 — is a short string, and only 1,930 distinct ones ever occur in Rook's world. Add one marker token, ⟨game⟩, for “a new game begins”, and you have a vocabulary of 1,931. A game is a sentence written in it. The game of Chapter 5 — predict the next token, score it with cross-entropy, descend — carries over without changing a comma.

Chess, second, because every claim is checkable. A move is legal in a position or it is not, and a small library of rules (this page carries chess.js, a complete referee) settles the question instantly, every time. Nothing in the vocabulary above knows this: the same three tokens Rook can emit at any moment are, in one particular position, two moves and one piece of nonsense.

That gap is what the V means in RLVR — reinforcement learning from verifiable rewards1 Lambert et al. · 2024 Tülu 3: Pushing Frontiers in Open Language Model Post-Training arXiv preprint · arXiv:2411.15124: the judge is not anyone's taste but a verifier that cannot be flattered, bribed, or fooled by confident nonsense. When frontier models are trained to reason about mathematics and code, this is the same move: reward what can be checked.2 Guo et al. · 2025 DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning arXiv preprint · arXiv:2501.12948

One more thing, and it shapes everything that follows: Rook never sees a board. No 8×8 grid, no piece list, no rules — only text, one move-token after another, 128 plies of context at most. Everything it appears to know about squares and pieces has to be squeezed out of the statistics of move strings. Keep that in mind as you watch it play: the little diagrams on this page are for you; the model gets the sentence.

Meet the student, then: four transformer blocks, 128 numbers wide, four attention heads — 1.3 million parameters. The same weights thread through this entire chapter; what pretraining builds is exactly what fine-tuning bends and reinforcement sharpens. The plates need WebGPU (a current Chrome or Edge) and wake on their own as you reach them — the whole chapter shares one model, running on your GPU.

What did pretraining buy? At step 0 the time machine shows you noise: roughly one sampled move in a hundred is legal, and that one by accident. By step 2,600, about 87 in a hundred are legal — played in positions the model has never seen, under rules nobody stated. The only pressure ever applied was next-token prediction over some six thousand games. Legality condensed out of statistics, the way grammar condenses out of text: not because the loss asked for it, but because predicting the corpus well is impossible without it.

Legality was not the only thing that condensed, and the rest can be checked rather than argued about.3 Toshniwal et al. · 2022 Chess as a Testbed for Language Model State Tracking AAAI 2022 · arXiv:2102.13249 Fit the simplest readout you can build — one linear layer, no bend — to the numbers running through a trained model, and ask it where the pieces are. They are in there, more clearly with every block. In the cleaner case of Othello the board comes out sharply enough to edit: change it inside the activations, let the model carry on, and its moves follow the board you planted rather than the one its own moves implied.4 Li et al. · 2023 Emergent World Representations: Exploring a Sequence Model Trained on a Synthetic Task ICLR 2023 · arXiv:2210.13382 That first result needed a probe with a bend in it, and what fixed it was the question rather than the model — ask each square black or white and a straight readout struggles; ask it mine or theirs and one works fine.5 Nanda, Lee & Wattenberg · 2023 Emergent Linear Representations in World Models of Self-Supervised Sequence Models arXiv preprint · arXiv:2309.00941 Which is why the plate below is drawn from the side to move's point of view. On chess models trained the way Rook is, the readout recovers more than the position — it recovers an estimate of how strong the player being imitated was.6 Karvonen · 2024 Emergent World Models and Latent Variable Estimation in Chess-Playing Language Models COLM 2024 · arXiv:2403.15498

None of which is proof of understanding, and the plate above is careful about the difference. A probe that scores well might be reading the model, or it might be reading the moves that were fed into it — which is what the control line measures, and why it sits above the dashed one rather than on it. The claim the experiment actually supports is narrower and still remarkable: a machine that has only ever seen sentences ends up carrying something shaped like the thing the sentences are about.

Now look at what pretraining did not buy. Read the sampled games: pieces drift, nothing is hunted, material hangs unpunished. Of course it does — the corpus is random legal play, and imitation is faithful to a fault. Rook plays like its diet, because its diet is the only definition of “good move” it has ever been shown. The fix is not more data. It is better data. Before we administer it, sit across the board from your student and feel what 87% legal actually plays like.

Better data, same machine

To see why the second act exists, look at what the first one actually produces. A pretrained model is a completion machine: ask a raw base model a question and it may answer — or continue with three more questions, or a survey form, because that is what question-shaped text does in its corpus. The assistant on the other end of a chat window is made from one by supervised fine-tuning7 Ouyang et al. · 2022 Training Language Models to Follow Instructions with Human Feedback NeurIPS 2022 · arXiv:2203.02155: continued training on a curated set of (instruction → good response) pairs, written and vetted by people. It adds little knowledge. What it teaches is the format of being helpful — that in this house, a question is followed by an answer. One pair from such a corpus looks like this:

Mechanically, nothing changes: the same weights, the same cross-entropy, the same update $\htmlClass{eq-model}{\theta} \leftarrow \htmlClass{eq-model}{\theta} - \htmlClass{eq-knob}{\eta} \htmlClass{eq-world}{\nabla \mathcal{L}}$ — only the corpus is now chosen on purpose. For Rook the mapping is direct. The “instruction” is the game so far; the “good response” is what a competent player did next. Where a frontier lab curates tens of thousands of demonstrations, we curate 2,381 games played by a greedy little bot that grabs material whenever it can, and half the time gives check when it cannot. Its style is loud: 38% of its moves are captures, against 12% in the random-play corpus underneath, and one game in twenty ends in checkmate rather than one in seventy.

One paragraph of honesty before the button. Fine-tuning shifts style, and it charges a price: as the weights lean toward greedy play, the old random-play corpus becomes more surprising to the model, and its validation loss there quietly rises. That drift is not a malfunction — it is what specialization looks like from the old distribution's point of view. It is also why fine-tuning is run at a smaller step size than pretraining (we ease $\htmlClass{eq-knob}{\eta}$ from 1.2·10⁻³ down to 3·10⁻⁴): nudge the weights and the style shifts; blast them and the old competence goes too. The plate below measures both sides of the trade, because a curve that only shows the win is an advertisement, not an experiment.

A judge instead of examples

Before moving on, scroll back and play your fine-tuned student — Plate IV always drives the current weights, and you will feel the difference: it comes for your pieces now.

The last act needs no examples at all. It needs a judge. Here is one full iteration of RLVR, as this page runs it: take a real opening from the corpus — four to ten plies, different every time — and have Rook finish it $G = 8$ different ways, up to fourteen plies each, sampled hot enough to disagree with itself. Then chess.js replays every continuation from that exact position and issues a verdict no one can argue with. Rollout $i$ earns

where $\htmlClass{eq-world}{\ell_i}$ counts the consecutive legal plies before the first illegal one, $\htmlClass{eq-world}{n_i}$ counts the plies attempted, and the bracket pays a half-point bonus only when the whole rollout survives the judge. Play twelve clean plies out of fourteen and you score well; break on ply two and you score badly; stay perfect and you are paid extra.

Raw rewards are not used directly. Within the group of eight, each reward is compared to the others:

This $\hat{A}_i$ is the advantage: how much better or worse rollout $i$ did than its own siblings. Chapter 6 stabilized learning by measuring outcomes against a baseline instead of trusting raw reward; this is the same idea with the scaffolding removed — the group is the baseline. That trick — the heart of the GRPO family of methods8 Shao et al. · 2024 DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models arXiv preprint · arXiv:2402.03300 — needs no second network to estimate values: eight siblings, sampled from the same position, are estimate enough.

The update is REINFORCE, exactly as the policy-gradient chapter promised it would be:

— raise the log-probability of every token in the above-average rollouts, lower it for the below-average ones, and touch only the generated tokens: the prefix was given, so it gets no credit. And one honest edge case, which you will see happen: if all eight rollouts earn the same reward, the standard deviation is zero, the advantages are undefined, and the step is skipped. When every answer is equally good, there is no gradient — reinforcement learning learns from differences, and a group of equals teaches nothing.

Now the two things this update does not have, because chapter 7 just spent a chapter installing both, and a plate that quietly omits them would be teaching you the wrong lesson. There is no importance ratio and no clip. That is not a simplification — it is a consequence: we take exactly one gradient step per group of rollouts, so the policy that generated the data is the policy being updated, every ratio is 1, and a fence at 1 ± ε would never be touched. Clipping machinery is the price of reusing a batch, and this loop does not reuse one. Read any paper in this family with that in hand: how much clipping it carries is a direct readout of how many updates it takes per generation.

And there is no KL leash. A frontier run would carry one — the same $\htmlClass{eq-knob}{\beta}$ you slid a moment ago — pinning the policy to the model it started from. Here the leash's job is done by a very small step size instead, and the choice was forced rather than principled: at pretraining rates this policy collapses into repeated tokens within a dozen updates. That is measured, not feared. What you are watching is the cheapest thing in the family that still works, and knowing exactly which safety rails have been taken off is the difference between running a demo and reading one.

Line them up

You have now made three different chess players out of one set of weights, and the honest way to compare them is not a curve — it is a decision. The page has been quietly photographing your work: every time the fine-tuning or RLVR loop pauses, it snapshots the weights, so all three stages survive even though the resident model keeps moving on. The arena below puts them at the same board. Play a move, and every fielded Rook is asked the same question — what would you do here? — each answer drawn as an arrow in its stage's color, with the numbers that explain it: how much of its belief was even legal, and what it wanted most before the mask. Watch the pretrained arrow wander while the fine-tuned one aims at your pieces; watch the RLVR column hold its legal mass in positions that make the earlier stages guess.

Where the judge works today

The loop you just ran is the loop, at reduced scale. When today's models are trained to write code, the verifier is a bank of unit tests: the program compiles and passes, or it does not, and no amount of confident prose changes the verdict. That is a move being legal or not, wearing different clothes. In mathematics, the judge checks the final answer, or a proof checker walks the argument line by line. Same iteration, exactly: sample a group of attempts from one problem, let the verifier score each, standardize within the group, reinforce what held up. The graders got bigger; the gradient did not change.

This is also why the verifiable domains are the ones moving fastest, and you have already felt the other half of that sentence. A judge made of rules is cheap, instant, tireless, and incorruptible; you can ask it a million times a day and it never lowers its standards, so no curve of yours ever peaks and turns over. A judge made of taste is expensive, slow, noisy, and has holes an optimizer will find. So coding and mathematics improve at a pace that essay-writing and advice do not — the pipeline is identical, and only some subjects come with an answer key.

The caveat is that "no feelings to flatter" is not "no exploits": a model rewarded for passing tests will sometimes learn to game the tests — patching the assertions, special-casing the inputs, exiting early with the expected code — and building verifiers that cannot be lawyered is a live problem rather than a solved one. There is a nastier turn to it, too. Read the model's own reasoning and you can often catch it planning the exploit in plain language; train against that signal and it does not stop exploiting, it stops saying so.9 Baker et al. · 2025 Monitoring Reasoning Models for Misbehavior and the Risks of Promoting Obfuscation arXiv preprint · arXiv:2503.11926

There is a second consequence of grading on a curve, and it decides how a run is fed. A group only teaches when its members disagree: all eight legal and the advantages vanish, all eight hopeless and they vanish too. So a group-relative method learns only at the frontier of what the model can currently almost do — which makes the difficulty distribution of your data not a detail of the pipeline but arguably its most important hyperparameter. Feed it problems it has already mastered and you will burn a week of compute on text that teaches nothing, while the reward curve sits reassuringly pinned near the top. Neither number on your dashboard will look broken. The industrial fix is as blunt as it sounds: keep sampling a prompt until its group contains both successes and failures, and drop the prompts that never manage both.10 Yu et al. · 2025 DAPO: An Open-Source LLM Reinforcement Learning System at Scale arXiv preprint · arXiv:2503.14476

And notice how the three stages fit: reinforcement could not have started from noise — a group of rollouts that are all hopeless earns identical rewards, and as Plate VI showed, a group of equals has no gradient. Pretraining built enough competence for the judge to have something to grade; fine-tuning aimed it; only then could verified reward pull it the rest of the way. Each act feeds the next. That ordering is not a convention — it is the reason the pipeline has this shape.

Step back and look at what you ran. One set of weights was pretrained on everything available, fine-tuned on a curated slice, then reinforced against a verifier. Swap chess for language. Swap legality for human preference and verified answers. Swap 1.3 million parameters for a trillion, and a browser tab for a datacenter. The recipe you just ran is the recipe. The ideas do not change — only the appetite. And the appetite is not a hand-wave: this same text-only treatment of chess, given enough of both, plays grandmaster-level blitz without searching a single move ahead.11 Ruoss et al. · 2024 Amortized Planning with Large-Scale Transformers: A Case Study on Chess NeurIPS 2024 · arXiv:2402.04494

And that is every machine in this book that answers a question. A loss surface and a step downhill; a neuron, a bump of influence; space bent until classes come apart; handwriting read out of 784 numbers and no picture; a map that draws itself; a game of guess-the-next-word; a policy beating to windward with nothing to go on but a clock; a judge fitted to your own eye and then broken by an optimizer — and here, all of it at once, teaching a pocket of numbers to play a game whose rules nobody told it. None of it stayed mysterious once you could watch the gradients move. That was the point.

Two chapters are left, and they are about the other thing a machine can do with what it has learned. Everything so far was handed something and asked about it. What remains is handed nothing at all.