World models, from first principles

A complete learning path · mathematics, neural networks, prediction and action

How can a machine learn what will happen next—and use that knowledge to decide what to do? This course builds the answer from basic mathematics to Yann LeCun’s joint-embedding predictive architecture, or JEPA, and the research paper LeWorldModel (LeWM).

By the end: Explain why an image can omit information needed to predict.

A world model learns some aspect of how an environment changes. Here, a camera observes a robotic arm; a neural network turns its images into a compact description; another network predicts how that description will change after an action. A planner can then compare possible actions before moving the real arm.

We will build every part of that loop. The course explains the mathematical prerequisites along the way, with worked calculations, derivations and experiments. The browser model is small enough to inspect; the final chapters show how the same ideas lead to the larger system in LeWorldModel, version 1.

One mechanism, three questions

Our running world is a small two-link robotic arm seen by a camera. First ask: what can we know from an image? A picture reveals position but may hide motion. Two arms can look identical now while moving in opposite directions.

Then ask: what description should we learn? Keeping every pixel preserves shadows and table texture along with the arm. Keeping too little may erase the joint configuration. A representation is useful only if it retains distinctions that later prediction or action requires.

Finally ask: how can predictions choose an action? We can imagine several commands, compare their predicted consequences with a goal, execute one command, and look again. The observation after an action may correct the model’s expectation.

These questions stay with us throughout the book. Every new piece of mathematics will answer one of them, expose a failure, or let us test an answer.

The route through the book

The linked route follows the ideas each later chapter needs. It reaches a small trainable world model before the harder Gaussian theory and transformer architecture, so those abstractions have a concrete job when we return to the research paper.

How to use a mathematical lesson

Before looking at a worked answer, try the small calculation or prediction. If a symbol is unfamiliar, stop at its definition and substitute numbers. A formula should become a calculation you can perform, not a sentence you recognize.

Core derivations are visible in the reading path. Exercise answers can be expanded after an attempt; all answers are included in print. The interactive desks ask you to predict a change before moving a control. Later, the learning laboratory really updates neural-network weights. Its curves are measurements of your run, so they may differ from the recorded examples.

Our starting point is basic algebra, elementary derivatives and integrals, and an introductory encounter with vectors and probability. We rebuild the specific tools we need, including partial derivatives, covariance, and normality tests. The reference chapter supplies a symbol guide and a map from the final paper back to its explanations.

A first prediction

Imagine two camera frames that show the arm in exactly the same position. Is its next position necessarily the same in both cases?

Learning before labels

Before asking which loss to minimize, ask what kind of knowledge an agent needs. A useful prediction is a prediction about something, for a reason.

By the end: Distinguish a world model, a representation, and a preference.

The lesson hidden in an ordinary video

Imagine watching a ball roll behind a screen. You expect it to reappear on the other side. Nobody needs to attach a class label to each frame for the sequence to contain a lesson. The visible past constrains the hidden present and the possible future. A learner can make a prediction, wait, and compare that prediction with what happens.

That comparison supplies a learning signal: information that can change the learner’s adjustable parameters. “Unlabeled” is not “without structure.” Time, spatial continuity, different views of the same scene, and the consequences of actions all provide structure. Self-supervised learning constructs an input and a target from that structure. A withheld image patch is a target; the next camera frame is another. The learner is not told an answer by a human annotator, but it still has an objective.

A supervised classifier solves a narrower problem: given an observation, predict a supplied label. If all training labels say whether a cup is present, there is little direct incentive to preserve the cup’s velocity or whether it is supported. A predictive learner can encounter pressure to preserve these quantities because they affect what happens next. A prediction objective can also exploit a shortcut such as the identity of the room or the unchanging background.

The central question is therefore not whether labels are good or bad. It is whether the training problem rewards the distinctions we will need later. For our mechanism, those distinctions include pose, motion, and response to action. Later goals may change without changing the mechanism’s dynamics.

What LeCun proposes, and what remains a proposal

Yann LeCun’s 2022 position paper, version 0.9.2, organizes a research program around learning from observation, planning through a world model, and representing the world at several levels of abstraction. It is a proposed synthesis, not a report that all its components have already been trained together successfully.

Its six main responsibilities are worth separating carefully. Perception extracts a useful state description. A world model predicts how that description can evolve. Cost expresses preferences, including intrinsic costs and a learned critic that anticipates future costs. An actor selects actions, possibly by optimizing a sequence through the model. Short-term memory retains information that the current observation does not provide. A configurator adjusts the other components to the task and circumstances.

The diagram below introduces the functional loop, before we specify neural networks or losses.

Our running example is a small slice of this proposal. The browser learner has a visual encoder, a predictor, a hand-specified goal cost, and a planner. It has a short observation history, but not the proposed general memory system. It has no learned configurator, no learned intrinsic motivational system, and no hierarchy of trained world models. Keeping that boundary visible helps us understand exactly what LeWorldModel contributes.

LeCun distinguishes reactive action from deliberative action. A reactive policy is a function that directly maps available information to an action. A deliberative system first imagines consequences and searches for a good action. One can then teach a fast policy to imitate the planner. The expensive deliberation becomes training data for a cheaper response. The term amortization means spreading a one-time learning cost across many later uses: repeatedly solving a problem teaches a function to approximate its solution.

State, observation, and memory are different things

Let the physical state be $s_t$, the action be $\act_t$, and an observation be $\obs_t$. A deterministic toy environment can be described by two functions:

$$s_{t+1}=F(s_t,\act_t),\qquad \obs_t=O(s_t).$$

These are definitions of a state-transition rule and a sensor rule. They are not claims that every real environment is deterministic. The notation $O$ here names the observation function; it is unrelated to the computational-complexity notation introduced later.

Suppose $s_t=(q_t,v_t)$ records position and velocity, while the camera reveals only position. In a simple unit-time model, $q_{t+1}=q_t+v_t$. The two states $(0,1)$ and $(0,-1)$ produce the same current picture but different next positions. No function of that picture alone can predict both correctly. This is an information problem, not a failure that a larger neural network can necessarily fix.

Two pictures can help. If velocity is constant between frames separated by time $\Delta t$, then $q_t-q_{t-1}=v\Delta t$. Dividing both sides by $\Delta t$ gives $v=(q_t-q_{t-1})/\Delta t$. If acceleration or occlusion matters, more history or a different state estimator may be needed. History helps when the earlier observations contain information about the hidden quantities needed for prediction. Some hidden quantities may remain unobservable even from a long sequence.

A state description is Markov for a prediction problem when, given that state and the relevant action, earlier history supplies no additional information about the next state. In everyday terms, the description has retained the past information needed for this prediction. Probability will later let us express that statement as conditional independence. The state must already include whatever memory matters. A compact learned embedding may fail to have that property even if the physical simulator’s full state has it.

Why predict a representation?

A camera describes surfaces, lighting, shadows, textures, and geometry all at once. A decision may depend strongly on some of these and weakly on others. A map of a railway system deliberately loses street-level detail while preserving connections needed for route planning. A learned representation is also a selective description, though its coordinates need not have names we recognize.

The analogy has a limit. A transit map is designed around a known purpose. In representation learning, we ask an objective and data to shape the map. If the objective says only “make the future easy to predict,” a blank map is excellent: nothing changes on it. If it says only “retain all variation,” irrelevant camera noise may occupy much of the representation. JEPA needs a balance between predictability and information retention.

An invariance identifies observations that differ in a way we decide not to preserve. If changing a tablecloth leaves an embedding unchanged, the encoder is invariant to that change on those examples. An equivariance preserves a structured response: rotating an object causes a corresponding transformation of its representation. Invariance is appropriate for a nuisance; equivariance may be appropriate for a variable needed in control. Removing orientation from an arm representation would make some goals impossible to distinguish.

An encoder cannot create information missing from its input. It can organize information, suppress variations, and make some relationships easier for a later model to use. A surprising success must still be supported by some informative pattern in the observations or history.

A description can hide detail without resolving uncertainty

Our map can leave out the tablecloth because a different tablecloth does not change the arm’s motion. But removing that detail does not tell us whether an unseen person will push the arm. These are two different problems: deciding what to describe and representing what remains uncertain.

LeCun’s broader proposal permits several compatible futures and uses a compatibility score called an energy. Lower scores mean a better fit. We will give this a mathematical definition after learning probability and prediction losses. LeWorldModel, our destination, uses a deterministic predictor; it implements a narrower part of that proposal.

Planning needs preferences as well as predictions

A perfect description of what can happen does not specify what should happen. The same model can support moving toward a cup, moving away from it, or remaining still. A goal cost tells the planner which consequences to prefer.

This separates learning dynamics from solving a particular task. LeWorldModel learns from observation–action trajectories without reward labels, then uses a goal image at planning time. “Reward-free training” therefore does not mean preference-free behavior. The planner still receives an objective.

It also does not mean action-free data. Knowing which action preceded a transition is essential to the particular action-conditioned learning problem. A video of another agent may support visual prediction without telling us which motor command our own agent should issue. Inferring actions from video is a further research problem.

Hierarchies shorten the question

To visit a friend, one might first choose a route across a city, then a street crossing, then individual foot placements. Long-range plans use coarser descriptions and larger time steps. A hierarchical world model would predict at several abstraction levels, allowing high-level subgoals to constrain lower-level action plans.

If each low-level step has $k$ alternatives, enumerating $H$ steps requires $k^H$ sequences: multiply $k$ choices once for every step. A good hierarchy can reduce the effective search by reusing meaningful subplans. That is the motivation; there is no general theorem that an arbitrary learned hierarchy will do so. Bad abstractions can erase the constraints needed to make a high-level plan feasible.

LeCun’s proposal also emphasizes memory. Moving one cup changes only a small portion of the world. A system that updates a persistent description of that cup may avoid reconstructing an entire scene description at every moment. Attention, taught later, provides one differentiable way to retrieve relevant stored information. A short frame history is a modest precursor, not a complete implementation of that idea.

A first worked challenge

A learner predicts the next frame of a swinging arm. Its training videos always use a dark background. It succeeds on those videos and fails on a bright background. Which part of our story does that challenge?

The rest of the book makes these questions quantitative. First we need to understand what a vector can preserve, what a distribution says about uncertainty, and how an error changes the function that produced it.

A world in coordinates

A representation is a choice of distinctions. Linear algebra gives us a language for asking which distinctions survive.

By the end: Encode a tiny image and compute a distance and a projection.

From an image to a list of numbers

A grayscale image of height $H$ and width $W$ is an array of $HW$ intensity values. A color image has an additional channel index, often with three entries for red, green, and blue. Flattening merely places these numbers in a fixed order. It loses no information if the shape and ordering are known. A $2\times2$ image with rows $(1,0)$ and $(0,1)$ can become the vector $(1,0,0,1)^\top$. The transpose symbol $\top$ tells us to write that list as a column.

Write the observation column as $\obs$, with entries $o_1,o_2,o_3,o_4$. An encoder is a function $f$ that maps this long vector to a shorter description, which we call $\lat$. As a deliberately simple example, define

$$f(\obs)=\begin{pmatrix}o_1+o_2\\o_3+o_4\end{pmatrix}.$$

This encoder retains total intensity in each row. It cannot tell $(1,0,0,1)$ from $(0,1,1,0)$ because both become $(1,1)$. A reader can now answer precisely what the encoder loses: the left–right arrangement within each row. Saying it produces a two-dimensional “embedding” adds no guarantee of usefulness.

In deep learning, an embedding is a learned coordinate representation. An encoder may intentionally give several inputs the same coordinates; in mathematics, an injective embedding instead keeps distinct inputs distinct. The issue is whether inputs requiring different predictions or actions remain distinguishable.

Matrices are coordinated linear measurements

The row-sum encoder is multiplication by

$$W=\begin{pmatrix}1&1&0&0\\0&0&1&1\end{pmatrix},\qquad \lat=W\obs.$$

For the first row, calculate $1o_1+1o_2+0o_3+0o_4=o_1+o_2$. The second row selects the other two pixels. The compact rule is $(W\obs)_i=\sum_j W_{ij}o_j$: $W_{ij}$ is the entry in row $i$, column $j$, and $\sum_j$ means add one term for every input coordinate $j$. Each row chooses a weighted measurement of the input. Matrix multiplication composes these measurements: if $y=A\lat$ and $\lat=W\obs$, then $y=(AW)\obs$. To verify this, expand $y_i=\sum_k A_{ik}\sum_j W_{kj}o_j$, exchange the finite sums, and collect the coefficient $\sum_k A_{ik}W_{kj}$ multiplying $o_j$. That coefficient is exactly $(AW)_{ij}$.

For example, let $A=(1,-1)$ subtract the second row sum from the first. Multiplication gives $AW=(1,1,-1,-1)$: applying the two maps successively gives the same answer as this single four-pixel measurement.

A neural network repeatedly applies such maps, offsets them by a bias vector, and inserts nonlinear functions between them. Without those nonlinearities, all the matrices would combine into a single matrix, no matter how many layers we stacked.

A tensor in this book is a multidimensional array with named axes. A batch of $B$ sequences, each containing $T$ color images, has shape $B\times T\times C\times H\times W$. An embedding tensor might have shape $B\times T\times d$. Exchanging axes preserves the values while changing which groups an average combines. Averaging over examples answers a different question from averaging over time.

Reading shapes before calculating

The notation $\mathbb R^d$ means lists of $d$ real numbers. A matrix in $\mathbb R^{m\times n}$ has $m$ rows and $n$ columns. Multiplying it by an $n$-entry column gives an $m$-entry column. Each output has one weighted sum with $n$ terms. For our encoder, $m=2$ and $n=4$.

Transpose exchanges rows and columns. Thus a column $x$ becomes a row $x^\top$. There are two different products to keep apart. If $x=(1,2)^\top$ and $y=(3,4)^\top$, then $x^\top y=1\cdot3+2\cdot4=11$, a scalar. But

$$xy^\top=\begin{pmatrix}3&4\\6&8\end{pmatrix},$$

a matrix whose $(i,j)$ entry is $x_i y_j$. This is the outer product. We will use it to record how coordinates vary together. Multiplication order changes both meaning and shape.

The identity matrix $I$ has ones on its diagonal and zeros elsewhere, so $Ix=x$. A square matrix is invertible when a map $W^{-1}$ undoes it: $W^{-1}W=WW^{-1}=I$. Our two-row, four-column encoder cannot have such an inverse, because different images give the same output. A linear combination adds scaled vectors. Vectors are independent if no nonzero choice of coefficients makes that combination zero; this will make the definition of rank concrete.

Length, angle, and projection

For vectors $x,y\in\mathbb R^d$, define the dot product $x^\top y=\sum_jx_jy_j$ and Euclidean length $\|x\|=\sqrt{x^\top x}$. For $x=(3,4)$ the length is $5$, by the Pythagorean theorem. The squared distance $\|x-y\|^2$ sums coordinatewise squared differences.

A unit vector $u$ has $\|u\|=1$. The scalar $h=u^\top x$ is the signed coordinate of $x$ along direction $u$. Why? Among points $cu$ on the line through $u$, minimize the distance to $x$:

$$\|x-cu\|^2=\|x\|^2-2c\,u^\top x+c^2.$$

Complete the square instead of guessing the closest point:

$$\|x-cu\|^2=(c-u^\top x)^2+\|x\|^2-(u^\top x)^2.$$

Only the first term changes with $c$. A square cannot be negative and is zero precisely at $c=u^\top x$. This proves the minimizing value without needing calculus. The projection vector is $(u^\top x)u$, while the projected scalar is $u^\top x$. SIGReg uses the scalar, one value per example and direction.

For $x=(3,4)$ and $u=(1,1)/\sqrt2$, the scalar projection is $7/\sqrt2$. A direction can reveal a relation between coordinates that neither coordinate alone reveals. We will reuse this diagonal measurement when comparing clouds of learned representations.

The same expansion proves the Cauchy–Schwarz inequality. The minimum of $\|x-cu\|^2$ is nonnegative, hence $(u^\top x)^2\leq\|x\|^2$. Taking $u=y/\|y\|$ for nonzero $y$ gives $|x^\top y|\leq\|x\|\|y\|$. Equality means one vector lies along the other. Thus the normalized dot product is between $-1$ and $1$ and can consistently be interpreted as the cosine of their angle. We will reuse this bound when comparing directions of motion.

Rank measures available directions

A linear map’s rank is the number of independent output directions it can produce. Its null space is the set of input changes it maps to zero. For the row-sum encoder, the change $(1,-1,0,0)$ lies in the null space. Moving brightness from one pixel to the other leaves the representation unchanged.

If all learned embeddings lie on one line, they may vary substantially in magnitude while using only one direction. That is dimensional collapse. If all are identical, even that one degree of variation disappears. Checking the average norm alone would miss both a constant nonzero cloud and many lower-dimensional failures.

Work one complete encoding by hand

Take an image with flattened pixels $\obs=(2,1,0,3)^\top$. Before continuing, compute its two row sums, their distance from the representation of $(1,2,2,1)^\top$, and their projection on $u=(1,0)^\top$.

We can now write descriptions and compare them. The next question is why the same description may accompany different futures. That requires probability; only afterward will we describe the geometry of a whole random cloud.

Prediction under uncertainty

A prediction can be wrong because the model is poor, because the observation is incomplete, or because several futures remain possible. Probability lets us separate these questions.

By the end: Compute an expectation and explain why squared error predicts a mean.

Random variables are measurements of uncertain outcomes

A random variable is a rule that assigns a numerical value to an outcome. “The next position of the arm tip” is one such measurement. A distribution specifies how probability is allocated to its possible values. The randomness may describe repeated experiments, variation across a dataset, or uncertainty conditional on what we currently know.

For a discrete variable, probabilities $p_i=P(X=x_i)$ are nonnegative and sum to one. For a continuous variable with density $p(x)$, probabilities belong to intervals or regions: $P(a\leq X\leq b)=\int_a^b p(x)\,dx$. The height $p(x)$ is not itself the probability of exactly $x$. Densities can exceed one; an interval of width $1/10$ with constant density 10 still has total probability one.

An embedding distribution arises when observations vary and the encoder maps them to vectors. The encoder can be deterministic while its outputs are random because its inputs are sampled. That is the distribution SIGReg shapes. It is different from a model’s distribution over alternative futures given one fixed observation.

Expectation is a weighted average

For discrete outcomes, define $\mathbb E[X]=\sum_i p_i x_i$. For a density, replace the weighted sum by $\int xp(x)\,dx$, when the integral is well defined. Expectation is linear because sums and integrals are linear:

$$\mathbb E[aX+bY]=a\mathbb E[X]+b\mathbb E[Y].$$

Independence is not needed for this identity. It is needed for a different one: if $X,Y$ are independent with suitable finite expectations, then $\mathbb E[XY]=\mathbb E[X]\mathbb E[Y]$. Here independence means that learning one variable does not change the distribution of the other. For discrete variables it says that the joint mass $p_{ij}=P(X=x_i,Y=y_j)$ factors as $p_iq_j$, where $p_i=P(X=x_i)$ and $q_j=P(Y=y_j)$. Then $\sum_{ij}x_iy_jp_iq_j=(\sum_ix_ip_i)(\sum_jy_jq_j)$. The integral proof is the same factorization.

Define variance by $\operatorname{Var}(X)=\mathbb E[(X-\mu)^2]$, where $\mu=\mathbb E[X]$. Expanding the square gives

$$\operatorname{Var}(X)=\mathbb E[X^2]-\mu^2.$$

Variance measures squared spread, so it has squared units. Standard deviation, its nonnegative square root, has the original units. A random variable may have a mean but infinite variance; our later variance calculations assume the required moments exist.

For two scalar variables, define $\operatorname{Cov}(X,Y)=\mathbb E[(X-\mathbb EX)(Y-\mathbb EY)]$. Expanding gives $\mathbb E[XY]-\mathbb EX\mathbb EY$. Independence makes this zero by the product-expectation identity above. Expanding the squared centered sum also gives $\operatorname{Var}(X+Y)=\operatorname{Var}(X)+2\operatorname{Cov}(X,Y)+\operatorname{Var}(Y)$. The same expansion works for any finite sum.

For independent identically distributed samples $X_1,\ldots,X_B$ with variance $\sigma^2$, let $\bar X=B^{-1}\sum_iX_i$. Then

$$\operatorname{Var}(\bar X)=\frac1{B^2}\sum_{i,j}\operatorname{Cov}(X_i,X_j)=\frac{\sigma^2}{B}.$$

The off-diagonal covariances vanish by independence, leaving $B$ diagonal terms. Thus quadrupling an independent sample size halves the standard deviation of the mean. Adjacent video frames are usually correlated. Treating them as independent can make an uncertainty estimate too optimistic.

Conditioning changes which average is relevant

Conditional probability restricts the experiment to outcomes compatible with known information. For events with $P(A)>0$, define $P(B\mid A)=P(A\cap B)/P(A)$. Multiplying through yields the product rule. Writing the same joint probability in the opposite order yields Bayes’ rule:

$$P(A\mid B)=\frac{P(B\mid A)P(A)}{P(B)}.$$

It is an algebraic identity, not a special learning algorithm. If 10 of 100 arm states move right and a noisy sensor reports “right” for 8 of those but also for 18 of the remaining 90, a positive report indicates rightward motion with probability $8/(8+18)$, not $8/10$. The base frequency matters.

A conditional expectation $m(x)=\mathbb E[Y\mid X=x]$ is the mean appropriate after observing $X=x$. In practice we learn an approximation to that function from examples. If the same image can accompany different velocities, the conditional distribution of the next image can remain broad even in a deterministic simulator.

For discrete outcomes, that conditional mean is $\sum_y yP(Y=y\mid X=x)$. If, after one sensor reading, the two possible velocities are $-1$ and $+1$ with probabilities $1/4$ and $3/4$, the mean velocity is $(-1)/4+3/4=1/2$. It is an average of compatible possibilities, not necessarily an actual velocity.

For continuous measurements, the event $X=x$ itself has probability zero, so we cannot divide its event probability into another. Instead use a joint density $p_{X,Y}(x,y)$. Integrate out $y$ to obtain $p_X(x)=\int p_{X,Y}(x,y)\,dy$. Where $p_X(x)>0$, define $p(y\mid x)=p_{X,Y}(x,y)/p_X(x)$ and $m(x)=\int y p(y\mid x)\,dy$. This conditional density integrates to one because its numerator integrates to its denominator. It is also the limit of conditioning on increasingly narrow intervals around $x$ when the densities are continuous there.

Why squared error predicts a mean

Fix an input $x$ and choose a scalar prediction $c$. Write $Y-c=(Y-m)+(m-c)$ with $m=\mathbb E[Y\mid x]$. Squaring and taking the conditional expectation eliminates the cross term:

$$\mathbb E[(Y-c)^2\mid x]=\operatorname{Var}(Y\mid x)+(m-c)^2.$$

The first term does not depend on the prediction. The second is minimized at $c=m$. The vector version follows by summing this identity over coordinates. Thus squared error favors the conditional mean when the target representation is fixed and the model can express that mean. In JEPA the representation also changes during learning, so this statement must be applied conditionally on the chosen encoder rather than treated as a complete theory of joint training.

If $Y=-1$ or $+1$ with equal probability, the best squared-error prediction is zero, although zero never occurs. Its expected error is 1. Predicting either endpoint gives expected error 2. A mean can be the optimal answer to one loss and a physically impossible outcome. This explains blurred pixel predictions and also warns that a deterministic latent predictor can average incompatible latent futures.

The Gaussian family

A standard Gaussian has density

$$p(x)=\frac1{\sqrt{2\pi}}e^{-x^2/2}.$$

The exponential determines the shape and the leading constant normalizes the area. We derive the constant now so this density has no unexplained normalization.

Symmetry makes the mean zero. To derive its variance, note $p'(x)=-xp(x)$ and integrate by parts:

$$\begin{aligned}\int x^2p(x)\,dx&=-\int xp'(x)\,dx\\&=-[xp(x)]_{-\infty}^{\infty}+\int p(x)\,dx\\&=1.\end{aligned}$$

The boundary term vanishes because the Gaussian exponential decays faster than $|x|$ grows. Integration by parts itself follows by integrating the product derivative $(uv)'=u'v+uv'$ and rearranging. We use this elementary bridge repeatedly, always checking boundaries.

If $G$ is standard Gaussian, $X=\mu+\sigma G$ has mean $\mu$ and variance $\sigma^2$. For $\sigma>0$, its density is $p_X(x)=\sigma^{-1}p((x-\mu)/\sigma)$. The factor $1/\sigma$ compensates for stretching the horizontal axis: substituting $g=(x-\mu)/\sigma$ restores unit area. If $\sigma=0$, the variable is constant and has no ordinary density of this form.

Independence is stronger than zero covariance

Let $X$ be uniform on $[-1,1]$, and define $Y=X^2$. Symmetry gives $\mathbb E[X]=\mathbb E[X^3]=0$, so $\operatorname{Cov}(X,Y)=0$. Yet observing $X$ determines $Y$ exactly. They are not independent. This counterexample explains why removing off-diagonal covariance cannot eliminate every dependency.

We have proved only one direction: independence implies zero covariance. The counterexample disproves the converse. The next chapter will distinguish these properties for entire vectors.

A mixture is another useful counterexample. Choose a branch with a coin, then sample a narrow Gaussian around either $-2$ or $+2$. The result has two peaks. Its mean is zero, and its variance can be computed by conditioning, but those two numbers do not make it a Gaussian. The book’s distribution experiment compares mixtures, rings, lines, constants, and Gaussians so these distinctions remain visible.

Generalization is a distributional question

The training distribution describes examples the learner sees while fitting parameters. The evaluation distribution describes the cases on which we measure performance. A train–test split estimates transfer to fresh data only to the extent that the split represents the desired use.

Overlapping video windows share observations and hidden episode conditions. Randomly splitting windows may place nearly identical transitions on both sides. Splitting entire episodes reduces this leakage. Test transfer to unfamiliar backgrounds, action ranges, and mechanisms by introducing those distribution shifts explicitly at evaluation.

The data-collecting policy also matters. If it never applies a negative torque at a particular pose, a model can fit the dataset without learning that counterfactual. The logged actions determine which consequences the data can teach; testing a missing action requires new coverage.

Quantifying finite-sample uncertainty

Suppose a controller succeeds independently on each task with probability $p$. A success indicator is 1 with probability $p$ and 0 otherwise. Its mean is $p$ and variance is $p-p^2=p(1-p)$. The sample success rate over $n$ tasks therefore has variance $p(1-p)/n$. Replacing $p$ by the observed rate gives a rough standard-error estimate, useful away from the endpoints and for sufficiently large independent samples.

For 48 successes out of 50, the rate is $0.96$ and this estimated standard error is about $0.028$. The standard error describes sampling variability. A normal approximation interval multiplies it by a chosen quantile; near a boundary such as 100% success, that approximation behaves poorly. For small experiments, report the count as well as any interval, and specify how tasks and seeds were sampled.

Variation across training seeds is another level of randomness. Testing three trained models on the same 50 tasks produces correlated evidence, not 150 wholly independent draws of a model–task pair. Separate variation from training, task selection, and evaluation randomness when possible. A plus-minus sign in a table is incomplete unless it says whether it denotes standard deviation, standard error, variance, or a confidence interval.

Worked exercises: choose the right average

A one-dimensional future is $-2$ with probability $1/4$ and $2$ with probability $3/4$. Find the best constant squared-error prediction and its irreducible error. Then explain what changes if a new sensor perfectly reveals the branch.

We can now distinguish sample fluctuations, hidden state, and genuine ambiguity. Next we apply these averages to vectors. Covariance will become a picture of which directions a cloud occupies, not merely a new matrix to memorize.

The geometry of a cloud

One vector describes one observation. A collection of vectors reveals what the encoder distinguishes across observations.

By the end: Compute covariance, inspect principal directions, and explain why whitening is not Gaussianity.

We now combine the vector operations from geometry with expectation and variance from probability. The question is concrete: if a collection of descriptions varies, does it vary in every useful direction, or only along a line?

Mean and covariance as geometry

For a batch $x_1,\ldots,x_B$, define the mean $\bar x=B^{-1}\sum_i x_i$. It is the point minimizing total squared distance to the samples. To see this, write $x_i-c=(x_i-\bar x)+(\bar x-c)$. On expanding the squares, the cross term vanishes because $\sum_i(x_i-\bar x)=0$. The remaining expression is

$$\sum_i\|x_i-c\|^2=\sum_i\|x_i-\bar x\|^2+B\|\bar x-c\|^2.$$

Only the last term depends on $c$, and its minimum is zero at $c=\bar x$.

Let $X_c$ be the matrix with centered row $x_i-\bar x$ for each example. Using a denominator $B$ for descriptive geometry, define

$$C=\frac1B\sum_i(x_i-\bar x)(x_i-\bar x)^\top=\frac1B X_c^\top X_c.$$

For a random vector, the population counterpart is $\Sigma=\mathbb E[(X-\mu)(X-\mu)^\top]$, with $\mu=\mathbb EX$. It averages over the distribution rather than a particular batch. The same coordinate and transformation rules apply to both.

Entry $C_{jj}$ is the average squared deviation of coordinate $j$. Entry $C_{jk}$ is the average product of deviations of coordinates $j$ and $k$. If they tend to rise and fall together, their covariance is positive. If one rises while the other falls, it is negative. Zero covariance means this particular linear co-variation vanishes; it need not mean independence.

For any direction $u$, multiplication gives

$$u^\top Cu=\frac1B\sum_i\big[u^\top(x_i-\bar x)\big]^2.$$

The right side is precisely the variance along that direction. It is nonnegative, so covariance matrices are positive semidefinite: their quadratic form is never negative. This phrase is a compact statement about projected variances, not a new kind of probability.

A statistical estimate of population covariance often uses $B-1$ instead. Here is the reason in one dimension. Let independent samples have mean $\mu$ and variance $\sigma^2$. The identity $\sum_i(X_i-\bar X)^2=\sum_i(X_i-\mu)^2-B(\bar X-\mu)^2$ follows by the same square expansion. Taking expectations gives $B\sigma^2-B(\sigma^2/B)=(B-1)\sigma^2$. Dividing by $B-1$ removes this particular estimation bias. We used the variance-of-the-mean identity proved in the preceding chapter. Neither denominator is universally “correct”; the objective must specify which it uses.

A centered batch has a rank limit

Subtract the mean from a batch of $B$ vectors to form the centered matrix $X_c$, with one example per row. Its rows sum to zero. At most $B-1$ rows can therefore be independent: the last is the negative sum of the others. Consequently,

$$\operatorname{rank}(X_c)\leq\min(d,B-1).$$

This is a structural limit, not a training defect. With $B=32$ and $d=192$, no single centered batch can have rank 192. This bound concerns the particular batch. The population covariance can have full rank even though each small centered batch is rank-deficient.

Eigenvectors reveal the cloud’s principal directions

An eigenvector $u$ of a matrix $C$ is a nonzero vector for which $Cu=\lambda u$. The matrix stretches that direction by a scalar $\lambda$ without changing its direction. For a covariance matrix, a unit eigenvector has projected variance $u^\top Cu=\lambda$.

Start with a matrix we can multiply by hand:

$$\begin{aligned}&C=\begin{pmatrix}2&1\\1&2\end{pmatrix}, \\ &u=\frac1{\sqrt2}\begin{pmatrix}1\\1\end{pmatrix}, \\ &v=\frac1{\sqrt2}\begin{pmatrix}1\\-1\end{pmatrix}.\end{aligned}$$

Compute $Cu=(3,3)^\top/\sqrt2=3u$ and $Cv=(1,-1)^\top/\sqrt2=v$. Thus the rising diagonal has variance 3 and the falling diagonal variance 1. Check also $u^\top v=0$ and both lengths equal 1: the two measuring directions are perpendicular and normalized.

For any unit direction $w=au+bv$, perpendicularity gives $a^2+b^2=1$. Its variance is $w^\top Cw=3a^2+b^2=1+2a^2$, between 1 and 3. This proves that the two displayed eigenvectors really are the least- and greatest-spread directions in this example.

Put these two vectors into the columns of a matrix $Q$ and their variances into a diagonal matrix $\Lambda$. Then

$$\begin{aligned}&Q=\frac1{\sqrt2}\begin{pmatrix}1&1\\1&-1\end{pmatrix}, \\ &\Lambda=\begin{pmatrix}3&0\\0&1\end{pmatrix}, \\ &C=Q\Lambda Q^\top.\end{aligned}$$

Verify the last equality by multiplication. Reading it right to left, $Q^\top$ expresses a vector along the principal directions, $\Lambda$ scales those coordinates, and $Q$ returns to the original coordinates. Here $Q^\top Q=I$, so transpose also undoes the rotation or reflection. A matrix with this property is orthogonal.

Why should two distinct eigenvector directions of a symmetric matrix be perpendicular? If $Cu=\lambda u$ and $Cv=\rho v$, then $u^\top Cv=\rho u^\top v$, but symmetry also gives $u^\top Cv=(Cu)^\top v=\lambda u^\top v$. Thus $(\rho-\lambda)u^\top v=0$. If the eigenvalues differ, the dot product must be zero.

For this two-dimensional example, we have explicitly found all the directions we need. The general statement that a symmetric matrix admits a complete perpendicular set is the spectral theorem. We will prove the finite-dimensional spectral theorem after learning directional derivatives. Until then, the calculations here use the displayed $Q$ that we can check directly.

The trace is the sum of diagonal entries: here $2+2=4$, also $3+1$. For any decomposition with orthogonal $Q$, expansion gives $\operatorname{tr}(Q\Lambda Q^\top)=\sum_j\lambda_j\sum_iQ_{ij}^2=\sum_j\lambda_j$. Each column’s squared entries sum to one. Total variance is unchanged by a perpendicular change of coordinates.

A Gaussian cloud in several dimensions

A multivariate standard Gaussian is a vector of independent standard Gaussian coordinates. Its joint density is the product of coordinate densities:

$$p(x)=(2\pi)^{-d/2}\exp(-\|x\|^2/2).$$

The exponent follows because multiplying exponentials adds their exponents. This density depends only on distance from zero. Rotating the coordinates with an orthogonal matrix preserves both length and volume, so it preserves the distribution. “Isotropic” means the distribution looks the same along every direction under these rotations; for this Gaussian the covariance is $I$.

Whitening is a moment operation

If all eigenvalues are positive, transform a centered vector by $y=\Lambda^{-1/2}Q^\top(x-\mu)$. Its covariance is

$$\operatorname{Cov}(y)=\Lambda^{-1/2}Q^\top C Q\Lambda^{-1/2}=I.$$

Each equality follows from applying the linear map to both sides of the covariance outer product and using $Q^\top Q=I$. Dividing a principal coordinate by the square root of its variance gives it unit variance. This operation is called whitening. If an eigenvalue is zero, its inverse square root does not exist; one must remove that direction or choose an explicit regularized approximation.

Whitening does not make every distribution Gaussian. Uniform points on a circle of radius $\sqrt2$ in two dimensions have mean zero and covariance $I$: symmetry gives equal coordinate variances, and their sum is the constant squared radius 2. Yet every sample lies exactly on a circle. A two-dimensional Gaussian fills an area and has variable radius. Matching first and second moments cannot distinguish these distributions. This example motivates SIGReg’s richer distributional measurements.

Worked exercises: what survived?

Take the three embeddings $(1,0)$, $(0,1)$, and $(-1,-1)$. Their mean is zero. Compute their descriptive covariance and identify one direction with the largest variance.

Our covariance measurements will later detect some forms of collapse. We have not yet learned how to change an encoder. The next two chapters build the calculus and learning rules that do that.

How a small change travels

Before we can train a network, we need to trace how changing one number changes another. We begin with a slope and finish with the matrix form of the chain rule.

By the end: Trace a derivative through a scalar chain and a vector-valued map.

We already know how to multiply matrices and measure squared distances. This chapter gives those operations sensitivities. Keep a concrete question in mind: if a prediction is too small, which adjustable number should move, and by how much?

Recover a derivative from a difference

For $f(x)=x^2$, change the input from $x$ to $x+h$. The output change is $(x+h)^2-x^2=2xh+h^2$. Divide by the input change:

$$\frac{f(x+h)-f(x)}h=2x+h.$$

As $h$ approaches zero, the ratio approaches $2x$. That limit is the derivative $f'(x)$. Near $x=3$, a small input change of $0.01$ produces approximately $6(0.01)=0.06$ output change. The exact change is $0.0601$; the omitted $h^2$ accounts for the difference.