Navigating the Deep Learning Landscape: Flat Minima and Generalisation — Epoche C2
The quantity in dispute The claim under examination is that the eigenvalues of the Hessian of the training loss, evaluated at the parameters a network has converged to, predict how far that network's test error will exceed its training error. Write $\hat L_S(\theta)$ for the empirical risk on a training sample $S$ of $n$ points, $L_{\mathcal D}(\theta)$ for the risk under the data distribution, and $H=\nabla^2\hat L_S(\hat\theta)$ for the Hessian at a converged point $\hat\theta$. A minimum is called sharp when the leading eigenvalues of $H$ are large and flat when they are small; the flat-minima hypothesis is that $L_{\mathcal D}(\hat\theta)-\hat L_S(\hat\theta)$ is small when they are small. Everything at issue in this essay is either an argument for that conditional, a measurement of it, or a demonstration that it cannot be stated in those terms without further qualification. One historical correction is needed before anything else. The hypothesis is not a recent revision of a prior belief that sharp minima fit the data more precisely and therefore generalise better; no such belief was held in the literature. Hochreiter and Schmidhuber proposed and defended flat minima in 1997, twenty years before the empirical work usually credited with the idea. What changed in the period from 2016 onwards is not the direction of the hypothesis but its evidential base — large-scale measurements linking sharpness to batch size, and optimisers designed to seek flatness deliberately. Why curvature should matter at all At a converged point the gradient vanishes, $\nabla\hat L_S(\hat\theta)=0$, so a second-order expansion loses its linear term: $$\hat L_S(\hat\theta+\delta)-\hat L_S(\hat\theta)=\tfrac{1}{2}\,\delta^{\top}H\delta+O\!\left(\lVert\delta\rVert^{3}\right)\;\le\;\tfrac{1}{2}\lambda_{\max}\lVert\delta\rVert^{2}+O\!\left(\lVert\delta\rVert^{3}\right).$$ The largest eigenvalue $\lambda_{\max}$ therefore controls the worst-case loss increase per unit of parameter displacement, and the trace $\operatorname{tr}H=\sum_i\lambda_i$ controls the average increase under isotropic displacement, since $\mathbb E\left[\delta^{\top}H\delta\right]=\sigma^{2}\operatorname{tr}H$ for $\delta\sim\mathcal N(0,\sigma^{2}I)$. That much is arithmetic. The substantive question is what the displacement $\delta$ is supposed to be. Here a statement that appears in most informal presentations of the subject has to be corrected, because it makes the argument incoherent. It is said that at inference time, noise or a shift in the data distribution "pushes the parameters out of the narrow valley". Nothing pushes the parameters anywhere. After training they are frozen; the input changes, not $\theta$. The displacement that matters is not in parameter space at all but between two surfaces over the same parameter space. Training minimises $\hat L_S$; what we care about is $L_{\mathcal D}$; and these are different functions, since $\hat L_S$ is a finite-sample estimate of $L_{\mathcal D}$. If their minimisers are separated by some $\Delta=\hat\theta-\theta^{*}_{\mathcal D}$, then, expanding $L_{\mathcal D}$ about its own minimiser, $$L_{\mathcal D}(\hat\theta)-L_{\mathcal D}(\theta^{*}_{\mathcal D})\approx\tfrac{1}{2}\Delta^{\top}H_{\mathcal D}\Delta.$$ For a given horizontal separation $\Delta$ — determined by sample size and by how much the empirical surface wobbles around the population surface — the vertical penalty scales with curvature. That is the whole mechanism, and it is the picture Keskar and colleagues draw explicitly: two surfaces slightly offset from each other, with a sharp minimiser paying heavily for the offset and a flat one paying little. A second correction is to the algebra. Flatness is sometimes written as $\hat L_S(\theta+\delta)\approx\hat L_S(\theta)+\epsilon$ for small $\epsilon$. This says nothing: $\epsilon$ is undefined, and any function whatever satisfies it for some $\epsilon$. The content of flatness is entirely in the rate — that the increase is $\tfrac12\delta^{\top}H\delta$ with small $\lambda_{\max}$ — and the informative statement is the quadratic one above. A third correction concerns the explanation usually attached. Sharp minima are said to have "memorised the training data, fitting even the noise". That cannot be the distinction, because in the over-parameterised regime both the sharp and the flat solutions typically interpolate the training set exactly, reaching indistinguishable training loss. The difference between them is not how well they fit but where in parameter space they sit while fitting equally well. The distinction is geometric, and describing it as a difference in fit conceals the only thing that is actually being claimed. The argument from description length, and its rigorous descendant Hochreiter and Schmidhuber's 1997 argument runs through coding rather than through perturbation. Define a flat minimum as a large connected region of weight space within which the loss stays close to its minimum value. If the acceptable region has half-width $\epsilon_i$ in coordinate direction $i$, then specifying a weight vector to the precision that matters costs about $\sum_i\log_2(R_i/\epsilon_i)$ bits, where $R_i$ is the prior range of that coordinate. Flat directions have large $\epsilon_i$ and cost few bits. Under the minimum description length principle, the total code length of model plus residuals is what should be minimised, so a flat minimum is preferred not by an appeal to robustness but by a count. Their algorithm implements this directly, adding to the loss a term that estimates the log-volume of the flat region from second derivatives of the network output with respect to the weights. The modern form of this argument is a PAC-Bayes bound, and it is the only route by which the flatness intuition has been turned into a theorem about test error. In one standard form: fix a prior $P$ over parameters before seeing the data, take any loss bounded in $[0,1]$, and let $Q$ be any distr