Deep Networks Rarely Get Trapped: The Geometry of High-Dimensional Loss Landscapes — Epoche C2
The claim, stated precisely enough to be wrong Training a deep network is the minimisation of a non-convex function of between $10^7$ and $10^{11}$ real parameters, and it succeeds far more reliably than non-convex minimisation has any right to: the training loss falls to near zero from a random start, on architectures nobody has analysed, with a recipe that is barely tuned to the problem. The explanation usually offered is that the landscape has few bad local minima, and that the rugged-terrain picture inherited from low-dimensional optimisation is simply the wrong picture in high dimension. That explanation is broadly right, and it is almost always stated in a form that is either unproved or false. Separating the two is the business of this essay. The slogan "most local minima are good" runs three logically independent claims together, and the evidence for them differs in kind. The counting claim. Among the critical points of the training loss — the parameter vectors at which the gradient vanishes — those whose loss sits well above the global minimum are overwhelmingly saddles rather than minima. The value claim. The local minima that do exist have losses concentrated in a narrow band just above the global minimum, so which one you land in barely matters. The dynamical claim. The trajectory that stochastic gradient descent actually follows terminates in that band, and in a part of it that generalises to unseen data. A landscape can satisfy the first two and still defeat an algorithm that spends its time in a region the counting argument does not describe; an algorithm can succeed in a landscape littered with bad minima simply by never going near them. Each claim therefore needs its own argument, and the arguments come from different literatures: the counting claim from the theory of random Gaussian fields on the sphere, the value claim from that theory plus exact results on linear networks, the dynamical claim from measurement and from convergence theorems specific to the over-parameterised regime. Taking them in that order, and marking where each runs out, is what follows. The spin-glass calculation, and exactly what it assumes The counting claim has one clean derivation, and it is worth seeing in full because its strength and its weakness are both in the assumptions rather than in the mathematics. Choromanska, Henaff, Mathieu, Ben Arous and LeCun (2015) observed that the output of a rectified-linear feedforward network can be written as a sum over input–output paths. A path through a network with $H$ hidden layers traverses $H+1$ weight matrices, so it contributes the product of the $H+1$ weights along it, multiplied by the input feeding it and by an indicator that equals one when every unit on the path is active and zero otherwise. The training loss is then a polynomial in the weights, homogeneous of degree $p = H+1$. To turn that polynomial into an object with a known landscape they impose three assumptions, which their paper states and labels as unrealistic. First, variable independence : the path inputs are treated as independent random variables. Second, redundancy : the number of distinct weights is far smaller than the number of paths, so the same weight recurs in many terms. Third, uniformity : every path is active with the same probability, independent of the weights. Add a spherical constraint on the weight vector and the loss becomes, up to constants, the Hamiltonian of a spherical $p$-spin glass — a homogeneous polynomial of degree $p$ in $N$ variables constrained to a sphere, with independent Gaussian coefficients. The third assumption is the one that hurts: making the activation pattern independent of the weights removes precisely the coupling that distinguishes a rectified-linear network from a random polynomial. Granting the reduction, the landscape is known exactly. Auffinger, Ben Arous and Černý (2013) computed the asymptotics of the number of critical points of each index — the index being the number of negative Hessian eigenvalues, so index zero means a local minimum. Their mechanism is the piece worth stating, because it explains where every constant comes from. At a critical point whose energy per variable is $u$, the Hessian on the sphere is distributed as a Gaussian orthogonal ensemble matrix shifted by $-pu$ times the identity. The GOE spectrum obeys Wigner's semicircle law, supported on a symmetric interval of radius $2\sqrt{p(p-1)}$ in the normalisation that makes the spin-glass constants come out. A critical point is a local minimum exactly when no eigenvalue is negative, that is, when the whole shifted semicircle clears zero: $$-pu \;\ge\; 2\sqrt{p(p-1)} \qquad\Longleftrightarrow\qquad u \;\le\; -\frac{2\sqrt{p(p-1)}}{p} \;=\; -2\sqrt{\frac{p-1}{p}} \;=:\; -E_\infty.$$ The factor $p$ in the shift is the degree; the radius carries $\sqrt{p(p-1)}$ because the Hessian of a degree-$p$ form involves two differentiations. Above the threshold $-E_\infty$, the shift no longer clears the semicircle, and the fraction of eigenvalues left below zero is a fixed positive number: a critical point above the threshold has an index proportional to $N$, so it is not merely a saddle but a saddle with order $N$ descent directions. The count of such points is exponentially large in $N$; the count of low-index points above the threshold is zero. This corrects the version of the claim that circulates, including in the earlier draft of this essay: it is not that the probability of a bad local minimum "decays exponentially with the dimension". It is that the local minima are confined to a band with hard edges, between the ground-state energy $-E_0$ and the threshold $-E_\infty$. The band is narrow. For $p=3$, corresponding to a network with two hidden layers, $E_\infty = 2\sqrt{2/3} = 1.633$, against a ground-state energy of about $1.657$ in the same units. The whole population of local minima is squeezed into an interval of width $0.024$, under $1.5$ per cent of the ground-state