More Parameters Than Data, and Still Generalising: The U-Shaped Curve Fails — Epoche B2
Revisiting Generalisation in Overparameterised Deep Learning Models The conventional wisdom in machine learning posits that increasing model complexity beyond a certain point inevitably degrades generalisation. This perspective is rooted in classical statistical learning theory [1] , which predicts a 'U-shaped' curve for the generalisation error as model capacity grows. On this view, 'overparameterised' models—those with more adjustable parameters than training data points—should merely memorise the training data, noise included, and so fail on unseen examples. Yet modern deep learning models [2] , often carrying millions or billions of parameters trained on far fewer data points, routinely achieve state-of-the-art performance, prompting a re-evaluation of the mechanisms governing generalisation. The Classical Bias-Variance Decomposition To understand the classical perspective, we begin with the bias-variance decomposition of the expected prediction error. Consider a true underlying function $f(x)$ that maps inputs $x$ to outputs $y$, corrupted by irreducible noise $\varepsilon$, such that $y = f(x) + \varepsilon$. We assume $E[\varepsilon] = 0$ and $\text{Var}[\varepsilon] = \sigma^2$, and that $\varepsilon$ is independent of the training set. Our goal is to train a model $\hat{f}(x)$ from a finite training dataset $D$ to approximate $f(x)$. The expected squared prediction error, also known as the generalisation error or risk $R(x)$, for a given input $x$ is defined as: $$ R(x) = E_{D,\varepsilon}\left[\left(y - \hat{f}(x)\right)^2\right] $$ where the expectation is taken over both the training datasets $D$ (from which $\hat{f}(x)$ is learned) and the irreducible noise $\varepsilon$. This error can be decomposed into three fundamental components: the squared bias, the variance, and the irreducible error. The cross terms vanish precisely because $E[\varepsilon] = 0$ and $\varepsilon$ is independent of $D$, leaving: $$ E_{D,\varepsilon}\left[\left(y - \hat{f}(x)\right)^2\right] = \left(E_D[\hat{f}(x)] - f(x)\right)^2 + E_D\left[\left(\hat{f}(x) - E_D[\hat{f}(x)]\right)^2\right] + \sigma^2 $$ Each term carries a distinct interpretation: The squared bias , $\left(E_D[\hat{f}(x)] - f(x)\right)^2$, measures the error due to the model's fundamental inability to capture the true underlying relationship $f(x)$. It is the squared difference between the average prediction of the model across all possible training datasets and the true function value. A high bias suggests the model is too simple or too constrained (underfitting). The variance , $E_D\left[\left(\hat{f}(x) - E_D[\hat{f}(x)]\right)^2\right]$, quantifies the variability of the model's predictions for a given $x$ across different training datasets. It reflects how sensitive the model is to fluctuations in the training data. A high variance indicates that the model is flexible enough to fit the training data, noise included, too closely, leading to poor generalisation (overfitting). The irreducible error , $\sigma^2$, is the variance of the noise $\varepsilon$ inherent in the data itself. This component cannot be reduced by any model, regardless of its complexity or training. The U-Shaped Risk Curve and Interpolation Threshold The bias-variance trade-off implies a characteristic relationship between model complexity and generalisation error. As a model's complexity (for instance, the number of parameters $p$, the polynomial degree, or the VC dimension) increases: Initially the model is too simple: high bias, because it cannot capture the nuances of $f(x)$, but low variance, because it is insensitive to small changes in the training data. The total error is dominated by bias. As complexity grows the model becomes more expressive and its bias falls, but its variance starts to rise as it becomes flexible enough to fit noise. Beyond an optimal point the bias keeps falling, possibly to zero if the model can fit the training data exactly, while the variance rises sharply, so the total generalisation error increases. This dynamic typically results in a 'U-shaped' curve for the generalisation error as a function of model complexity. For instance, a simplified model for the generalisation risk $R(p)$ as a function of the number of model parameters $p$ might exhibit this form: $$ R(p) \approx \frac{C_{\text{bias}}}{p} + C_{\text{variance}} \cdot p + \sigma^2 $$ where $C_{\text{bias}}$ and $C_{\text{variance}}$ are positive constants. The first term, representing bias, decreases with increasing $p$, while the second term, representing variance, increases with $p$. The minimum of this U-curve, at $p = \sqrt{C_{\text{bias}}/C_{\text{variance}}}$, represents the optimal balance between bias and variance, and thus the best generalisation performance the classical picture allows. A critical point on this curve is the interpolation threshold . This threshold is reached when the model's capacity, typically quantified by its effective number of parameters $p$, becomes comparable to the number of training samples $n$. At this point, the model gains sufficient flexibility to perfectly fit, or 'interpolate', the training data, achieving zero training error. Quantitatively, this condition is approximated by: $$ p \approx n $$ Before this threshold ($p \lt n$), models are typically underparameterised and cannot achieve zero training error. Beyond this threshold ($p \gt n$), models are overparameterised and can achieve zero training error in many different ways, raising concerns about overfitting and poor generalisation according to classical theory. Generalisation Bounds and Their Limitations Classical statistical learning theory also provides formal bounds on generalisation error in terms of a model's capacity and the number of training samples, quantifying the largest plausible difference between performance on the training data and expected performance on unseen data. The empirical risk , $R_{\text{train}}(f)$, is the average loss of a function $f$ on the tr