The Enigma of Deep Learning's Generalisation — Epoche C2
What is being claimed, and about what A convolutional network with more parameters than it has training images, trained until it fits every label exactly, will often have lower test error than the same architecture at a fraction of the width. The observation is robust, it is not what the textbook account of capacity control predicts, and the name it now goes by — double descent — was given to it by Belkin, Hsu, Ma and Mandal in 2019. This essay is about what that observation licenses one to say. The honest answer turns out to be narrower than the usual telling, and the narrowing is where the interesting content is. Start with the object the claim is against, because it is routinely misstated. For squared loss, at a fixed test input $x$, with the expectation taken over the random draw of the training set $D$ and the label noise, there is an identity: $$\mathbb{E}_{D,y}\big[(y - \hat f_D(x))^2\big] = \sigma^2 + \big(\mathbb{E}_D[\hat f_D(x)] - f^*(x)\big)^2 + \mathbb{E}_D\big[(\hat f_D(x) - \mathbb{E}_D[\hat f_D(x)])^2\big],$$ where $f^*$ is the regression function and $\sigma^2$ the noise variance. The three terms are the irreducible error, the squared bias and the variance. Two things about this are usually left implicit and both matter here. First, it is an identity, not a theorem about model classes: it says nothing about how bias and variance move as capacity changes, and it therefore cannot be falsified by any experiment. Second, it is specific to squared loss and to a fixed $x$; there is no comparably clean decomposition for the zero-one loss that classification experiments actually report. The trade-off is a separate, empirical claim laid on top of the identity: that as one enlarges the model class, bias falls and variance rises, so that test error is U-shaped in capacity. That claim is what double descent contradicts, and only that. The published version of this essay went further and said the phenomenon "decisively overturns" the bias–variance trade-off as an "immutable law"; by the end of what follows, that verdict will need qualifying in two directions at once. One correction before going on. The published version located the classical U-curve's upturn "after the interpolation threshold" — the interpolation threshold being the capacity at which the model first fits the training data exactly. That is not where the classical curve turns up. In the classical picture the minimum of test error sits well below interpolation, and error is already rising by the time the model has capacity to interpolate; the interpolation threshold is where the double-descent curve peaks , not where the classical one begins to climb. The two curves agree below the threshold and diverge above it, which is the entire point of the comparison. The evidence, and what was actually measured Three experimental results carry the weight, and they measure different things. Zhang, Bengio, Hardt, Recht and Vinyals (2017) took standard image architectures and trained them on CIFAR-10 with the labels replaced by uniformly random ones. The networks reached zero training error. Since the training set has fifty thousand images and ten classes, a network that has memorised random labels has test accuracy at chance, ten per cent, and test error ninety per cent — and the same architecture, on the true labels, generalises well. The consequence is structural rather than quantitative: the hypothesis class has enough capacity to fit any labelling of the training set, so its Rademacher complexity is essentially maximal, so any bound on test error that depends only on the capacity of the class is vacuous for these models. Whatever explains generalisation here cannot be a property of the class alone; it has to involve the data and the algorithm. Belkin and colleagues (2019) exhibited the curve itself in a setting where the capacity knob is unambiguous. They fitted random Fourier features to MNIST: draw $N$ random directions, form the corresponding features, and fit the labels by least squares, taking the minimum-norm solution when the system is underdetermined. Sweeping $N$ past the number of training points $n$, test error rises to a spike at $N = n$ and then falls again, going below its best value from the underparameterised regime. The location of the spike is the content of the experiment: it is not at some tuned capacity but exactly where the linear system stops being overdetermined. Nakkiran, Kaplun, Bansal, Yang, Barak and Sutskever (2020) generalised the curve away from any particular capacity parameter. Their construct is effective model complexity : for a training procedure and a data distribution, the largest sample size on which that procedure achieves approximately zero training error. Their hypothesis is that test error peaks when the effective model complexity is near the actual training set size, and this reorganises three phenomena into one. Model-wise double descent is what one sees varying width at fixed $n$. Epoch-wise double descent is what one sees varying training time at fixed width, since effective model complexity grows with training. Third, and omitted from the published version of this essay though it is half the title of the paper, is sample-wise non-monotonicity: for fixed model and training time, adding data can increase test error, because more data moves the ratio towards the peak rather than away from it. They also report the sharpening effect of label noise — the peak, faint on clean CIFAR-10, becomes pronounced when a fraction of the training labels are randomised. That is a clue about mechanism, and the next section says why. The one regime that is understood For linear and random-features models the whole curve can be computed, and doing so explains both the spike and the noise dependence. Consider least squares with design matrix $X \in \mathbb{R}^{n \times N}$. When $N $$\mathbb{E}\|\hat\beta - \beta^*\|^2 = \sigma^2 \operatorname{tr}\big((X^{\top}X)^{-1}\big) = \sigma^2 \sum_{i} \frac{1}{\