The Implicit Regularisation of Flat Minima in Deep Learning — Epoche C2
The fact that needs explaining A convolutional network can be trained to zero error on a copy of CIFAR-10 in which every label has been replaced by a uniform random draw from the ten classes, and the same architecture trained on the true labels reaches good test accuracy — a pair of facts established by Zhang and colleagues in 2017 and confirmed on ImageNet in the same paper. The two facts together are what any account of generalisation in deep learning has to accommodate, and they are more demanding than the usual statement of the puzzle. It is not merely that networks with more parameters than training examples generalise despite the classical warning about model complexity. It is that the same network, with the same optimiser and the same explicit regularisers, can memorise arbitrary labels. Whatever explains good generalisation therefore cannot be a property of the hypothesis class alone, because the hypothesis class is demonstrably rich enough to fit noise; it must be a property of which solution the training procedure selects out of the many that fit. The account examined here locates that selection in the geometry of the training loss surface: stochastic gradient descent, it is claimed, preferentially settles in flat minima rather than sharp ones, and flat minima generalise better. This essay reconstructs the argument in full, states the theorem that makes one version of it rigorous, and then sets out the objection — due to Dinh and colleagues in 2017 — that refutes the version in which the claim is usually stated. The earlier form of this essay presented the flat-minima story as settled; it is not, and the way it fails is more instructive than the story itself. What "flat" means, and why three definitions do not agree Throughout, $L(\mathbf{w})$ is the training loss as a function of the weight vector $\mathbf{w} \in \mathbb{R}^d$, and $\mathbf{H} = \nabla^2 L(\mathbf{w})$ is its Hessian in those coordinates. Near a minimum the second-order expansion is $$ L(\mathbf{w} + \Delta \mathbf{w}) \approx L(\mathbf{w}) + \nabla L(\mathbf{w})^{T} \Delta \mathbf{w} + \frac{1}{2} \Delta \mathbf{w}^{T} \mathbf{H}\, \Delta \mathbf{w} $$ and since $\nabla L(\mathbf{w}) = 0$ there, the increase in loss under a displacement is governed entirely by the quadratic term. A minimum is called sharp when $\mathbf{H}$ has large eigenvalues, so that a displacement of small norm $\|\Delta\mathbf{w}\|$ produces a large rise in $L$, and flat when they are small. So far this is the definition the earlier version gave, and it is a reasonable one. The difficulty is that at least three inequivalent quantities go by the name. The largest eigenvalue $\lambda_{\max}(\mathbf{H})$: the worst-case rise in loss per unit displacement, attained along the corresponding eigenvector. The trace $\mathrm{tr}(\mathbf{H}) = \sum_i \lambda_i$: the average rise under an isotropic random displacement. For $\epsilon \sim \mathcal{N}(0, \sigma^2 I)$ the expansion above gives $\mathbb{E}[L(\mathbf{w}+\epsilon)] \approx L(\mathbf{w}) + \tfrac{\sigma^2}{2}\mathrm{tr}(\mathbf{H})$, because the linear term vanishes in expectation and $\mathbb{E}[\epsilon^{T}\mathbf{H}\epsilon] = \sigma^2 \mathrm{tr}(\mathbf{H})$. Volume : the size of the connected region around $\mathbf{w}$ within which the loss stays below some threshold. This is the definition Hochreiter and Schmidhuber used in 1997, and it is worth being accurate about what they did, because the earlier version overstated it. They did not prove that flat minima generalise. They defined a flat minimum as a large connected region of weight space over which the error is approximately constant, argued on minimum-description-length grounds that such a region permits the weights to be specified with few bits, and derived from that argument an algorithm — Flat Minimum Search — that penalises the local sensitivity of the network's outputs to its weights. These come apart. A Hessian with one large eigenvalue and the rest near zero has large $\lambda_{\max}$ but small $\mathrm{tr}(\mathbf{H})/d$, and a very large low-loss volume in the remaining $d-1$ directions. Since the spectrum of $\mathbf{H}$ at a deep-network minimum is in fact strongly degenerate — a bulk concentrated near zero with a small number of large outliers — the choice among these three is not a technicality. The argument below will need the trace and the volume, not the largest eigenvalue. Why flatness should matter: the perturbation argument, done properly The earlier version justified the link to generalisation by saying that a model in a sharp minimum is sensitive to "noise in the test data or slight differences between training and test distributions". That reasoning does not go through, and the correction matters. The Hessian above is the curvature of the loss with respect to the weights . Sensitivity to perturbed inputs is the curvature with respect to the inputs , a different object entirely; a network can be flat in weight space and brittle in input space, and adversarial examples are the standard demonstration that the two are not tied together. The argument that does go through concerns the relation between two surfaces, not one. Training minimises the empirical loss $\hat{L}$ on a finite sample; what we care about is the population loss $L$. These are different functions of $\mathbf{w}$, and their minima do not coincide. If one models the discrepancy crudely, as the empirical surface being the population surface displaced by some vector $\mathbf{\delta}$ in weight space, then the excess population loss incurred at the empirical minimiser is $\tfrac{1}{2}\mathbf{\delta}^{T}\mathbf{H}\mathbf{\delta}$ — small when the curvature is small, in exactly the directions the displacement happens to lie. That is the honest intuition, and it is what Keskar and colleagues appeal to. It is an intuition and not a bound: it assumes the two surfaces differ by a translation, which is false. The rigorous version comes from PAC-Bayes ana