The Approximator

The prologue left you on a landscape. Every knob of a machine became an axis, wrongness became height, and learning became one small move repeated: measure the slope, step downhill. What it never said is what the machine is. This chapter builds one — the smallest one worth building — and then hands its knobs to gradient descent while you watch.

The working part is small enough to hold in your head. A neuron takes a number in, multiplies it by a weight $\htmlClass{eq-model}{w}$, adds a bias $\htmlClass{eq-model-2}{b}$, and passes the result through a fixed curve called an activation. This book starts with the gentlest activation, the hyperbolic tangent:

That is the whole organism. In one dimension its output is a smooth step: flat, a rise, flat again. The weight sets how steep the rise is and which way it faces; the bias slides it along the input. One step is a poor vocabulary — so we hire more. A layer is a row of neurons reading the same x, each with its own weight and bias: a palette of steps, tilted and placed differently. An output layer mixes the palette, scaling each step by an output weight and adding them all up:

Subtract one step from a slightly shifted copy and you get a bump. Sums of steps are therefore sums of bumps: little hills of influence you can place, widen, and flip. The universal approximation theorem makes this precise — with enough hidden neurons, f can match any continuous curve on an interval as closely as you like.1 Cybenko · 1989 Approximation by superpositions of a sigmoidal function Mathematics of Control, Signals and Systems 2 · web.njit.edu It holds from the other direction too: keep the layer narrow, barely wider than the input, and stack it deep enough, and you get the same guarantee.2 Lu et al. · 2017 The Expressive Power of Neural Networks: A View from the Width NeurIPS 2017 · arXiv:1709.02540

Read either of them honestly, though. They say the right weights exist; they say nothing about how to find them, and “enough” can be an absurd number. And a theorem about what a machine could represent is a weaker thing than it sounds — a large enough lookup table is a universal approximator too, and no use to anyone. The question that decides whether a network is worth building is not which curves it could draw in principle. It is which ones it finds easily, from a random start, by walking downhill. That is the previous chapter's business: write the error down as a loss, and send gradient descent looking.

One neuron, three knobs

Before training anything, get your hands on the unit itself. Below is the same neuron twice: on the left as a circuit — the ultramarine edge carries $\htmlClass{eq-model}{w}$, the violet edge carries $\htmlClass{eq-model-2}{b}$, and the blue-cyan edge on the way out carries the amplitude $\htmlClass{eq-model-3}{v}$, while the teal curve on the dial between them is whichever $\htmlClass{eq-op}{\sigma}$ is selected — and on the right as $\htmlClass{eq-model-3}{v}\,\htmlClass{eq-op}{\sigma}(\htmlClass{eq-model}{w}x + \htmlClass{eq-model-2}{b})$, the one shape it can draw. The third knob, $\htmlClass{eq-model-3}{v}$, is just another coefficient — an amplitude that stretches the bend taller or flips it upside down. Move each slider until you can predict what both views will do before you touch it. Then swap $\htmlClass{eq-op}{\sigma}$ itself and see which of your intuitions survive: the same three knobs pull very different shapes out of a relu than out of a tanh.

The bend is the whole point

Strip $\htmlClass{eq-op}{\sigma}$ out and a neuron is just $\htmlClass{eq-model}{w}x + \htmlClass{eq-model-2}{b}$ — a line. Stack a hundred all-linear layers and the stack collapses: a line of a line is still a line, so the deepest such network can be multiplied out into a single matrix wearing a hundred costumes. The activation is the only part of the machine that refuses to be linear, and everything a network can do that a line cannot — every curve, corner, and decision — is purchased at that little bend.

Training cares about a second, quieter property: the slope. Gradient descent reaches every weight through the chain rule, and the chain rule multiplies by $\htmlClass{eq-op}{\sigma'}$ at each layer it crosses on the way back. The field guide below therefore draws every activation twice — the function solid, its derivative dashed. Wherever the dashed curve hugs zero, learning goes quiet. The two classics saturate at both ends, which is how deep sigmoid networks starved for decades — the vanishing gradient, and a large part of why the field spent those decades shallow.3 Glorot & Bengio · 2010 Understanding the difficulty of training deep feedforward neural networks AISTATS 2010 · proceedings.mlr.press

relu4 Nair & Hinton · 2010 Rectified Linear Units Improve Restricted Boltzmann Machines ICML 2010 · cs.toronto.edu is silent across its entire left half instead, which sounds worse and is not: on the right it never saturates at all, so a gradient can cross ten layers without being multiplied down to nothing. The cost is that a unit driven far enough left stops receiving gradient and is called dead; its leaky cousin5 Maas, Hannun & Ng · 2013 Rectifier Nonlinearities Improve Neural Network Acoustic Models ICML 2013 WDLASL · ai.stanford.edu keeps a trickle flowing on purpose. That half of relu's units are silent at any moment turns out to be part of the appeal rather than a defect — the network computes on a changing, sparse subset of itself.6 Glorot, Bordes & Bengio · 2011 Deep Sparse Rectifier Neural Networks AISTATS 2011 · proceedings.mlr.press

The last three — gelu7 Hendrycks & Gimpel · 2016 Gaussian Error Linear Units (GELUs) arXiv preprint · arXiv:1606.08415, silu8 Elfwing, Uchibe & Doya · 2017 Sigmoid-Weighted Linear Units for Neural Network Function Approximation in Reinforcement Learning Neural Networks 107 · arXiv:1702.03118, mish9 Misra · 2019 Mish: A Self Regularized Non-Monotonic Activation Function BMVC 2020 · arXiv:1908.08681 — sand the corner off and keep a little slope slightly below zero, which is one reason the transformer era settled on them. Silu arrived twice: once from researchers building reinforcement learners, and again the same year, out of an automated search over candidate formulas that reported it as a discovery under a different name.10 Ramachandran, Zoph & Le · 2017 Searching for Activation Functions arXiv preprint · arXiv:1710.05941 That is roughly how this corner of the field works. Nobody derives an activation from first principles; they are found, measured, and kept if they win.

None of this changes what networks can express — the universal approximation theorem is not picky about the bend. What it changes is the handwriting of the fit and the temperament of training, and you will see both first-hand in the workshop that follows.

The curve workshop

Now give the knobs away. The workshop below holds a whole network and trains it live, in a worker beside this page, by the only rule this book uses — $\htmlClass{eq-model}{\theta} \leftarrow \htmlClass{eq-model}{\theta} - \htmlClass{eq-knob}{\eta} \htmlClass{eq-world}{\nabla \mathcal{L}}$ — with the loss set to the mean-squared error, the average of $(f(\htmlClass{eq-world}{x}) - \htmlClass{eq-world}{y})^2$ over the training points. Every step pulls the network’s curve a little closer to a target.

You see the machine three ways at once. On the left sits the network itself: every edge is one weight, drawn thicker as it grows, ultramarine when positive and vermilion when negative. On the right, the curve it currently draws against the dashed target. And underneath, the palette — the curves the deeper layers output, ending in the last hidden layer’s contributions, each drawn in the color of its output weight’s sign and thicker as that weight grows. Hover any neuron in the diagram to light up its wiring — and, if it owns a tile, the curve it outputs; or hover a tile to find its neuron.

Train on the sine and watch the palette organize — units that began as arbitrary steps drift into position, each claiming a stretch of the curve. Switch the target to bumps mid-training and the units renegotiate. Then take the pen and draw your own curve straight onto the plot: the network will chase it while you drag.

What width buys, what depth buys

Width buys vocabulary. Every extra hidden unit is one more step the output layer can place, so a wider palette affords finer detail: at width 2 the sine defeats the network — two steps cannot make three bends — while at width 16 it has bumps to spare. Watch the parameter count as you widen, though. Vocabulary is not free.

Depth buys reuse. With one hidden layer, every wiggle must be purchased with its own neurons. Add a second layer and the new neurons stop reading x directly: they read the first layer’s steps, and can bend an already-bent thing — bumps of bumps. A third layer bends the bumps of bumps again. The same budget of parameters goes further because pieces are reused instead of re-made.

That is not just a nicer story; it is a measurable gap. Count the straight pieces a relu network's fit is made of and the count grows roughly in proportion to width, but multiplies with each layer of depth — every layer folds the folds beneath it.11 Montúfar et al. · 2014 On the Number of Linear Regions of Deep Neural Networks NeurIPS 2014 · arXiv:1402.1869 And there are functions a deep network draws with a handful of units that no shallow network can match without an exponential number of them.12 Telgarsky · 2016 Benefits of depth in neural networks COLT 2016 · arXiv:1602.04485 In one dimension the difference is subtle; in the chapters ahead, where inputs are images, it is most of the story.

The activation sets the network’s handwriting. tanh is a soft wave, so its sums are smooth and rounded everywhere. The rectified linear unit — relu, zero on the left, a straight ramp on the right — creases instead of curves: its sums are piecewise-linear, and you can count the folds in the fit. Switch the workshop to relu on the |x| target and it lands almost at once, because the target is itself two creases. Switch back to tanh and watch it round a corner it can never make sharp. The workshop also carries the modern pair from the field guide — gelu and silu — which crease like relu with the corners sanded smooth; try them and read the difference straight off the palette.

One honest warning before you scale everything up. The workshop samples its target at 256 points packed densely along the interval, so fitting the data and fitting the curve are nearly the same task. They usually aren’t. With few points and many neurons, a network can pass through every training point exactly and still be wrong everywhere between them — wiggling where it should glide. That failure is called overfitting, and nothing in the loss ever asked the network to behave between the points. This book meets it properly once real data arrives.

The old lesson drawn from that is: capacity is dangerous, so keep the model small. The last decade made a mess of the lesson. Take a network that reads photographs well, shuffle the labels so every answer is now noise, and train it again — it fits all of them, perfectly, memorising the lot.13 Zhang et al. · 2017 Understanding deep learning requires rethinking generalization ICLR 2017 · arXiv:1611.03530 Its capacity to overfit is total. Hand back the real labels and the very same network generalizes. Whatever stops it from doing the same to those, it is not a shortage of room.

Stranger still is what happens if you keep growing it. Test error rises as the textbook promises, peaks right at the size where the model can just barely fit its training set exactly — and then, as the model gets bigger still, comes down again, often below anything the small models managed.14 Belkin et al. · 2019 Reconciling modern machine learning practice and the bias-variance trade-off PNAS 116(32) · arXiv:1812.11118 The curve has two descents, and the classical one is only the first. The models in the news live far out on the second, in the region the textbook picture calls hopeless, and the same double dip appears in how long you train, not only in how big you build.15 Nakkiran et al. · 2019 Deep Double Descent: Where Bigger Models and More Data Hurt ICLR 2020 · arXiv:1912.02292 None of this repeals overfitting. It moves the explanation somewhere more interesting: what keeps an over-parameterised network honest is not how few parameters it has, but which of the enormous number of perfect fits gradient descent happens to walk to.

This chapter happened in one dimension on purpose: you could see every neuron, every bump, and the whole sum at once. Nothing was hidden — the palette under the plot is the network, laid out flat.

The next chapter keeps the machine and changes the canvas. In two dimensions a network stops looking like a sum of bumps and starts doing something stranger and better: it bends the space the data lives in, until problems that looked hopeless become straight lines. When you’re ready, go bend space.