A Variance Perspective on Empirical Bayes — Epoche C1
The claim, and the competitor it must be tested against Estimating a prior distribution from the same data one then analyses — the manoeuvre that defines empirical Bayes — has a price, and this essay computes it exactly in the case where it can be computed. The claim usually made is that empirical Bayes methods, by adapting the prior to the data, always reduce estimation error relative to non-adaptive alternatives. The claim usually made against it is that the extra variability introduced by estimating the prior can outweigh the gain. The second claim is correct, but only against one of the two competitors it is normally aimed at, and getting that right is where the argument begins. Four terms fix the setting. Observations $X_1,\dots,X_n$ are drawn from distributions indexed by unknown parameters $\theta_1,\dots,\theta_n$, one per observation. A prior is a distribution $g(\theta \mid \eta)$ from which the $\theta_i$ are themselves supposed drawn, and $\eta$ is its hyperparameter — a parameter of the prior rather than of the data. The maximum likelihood estimator (MLE) of $\theta_i$ ignores the prior entirely and takes whatever value of $\theta_i$ makes $X_i$ most probable. A fixed-prior Bayes estimator uses the posterior mean $E[\theta_i \mid X_i, \eta]$ with $\eta$ known and specified in advance. Empirical Bayes, following Robbins (1956), replaces the unknown $\eta$ by an estimate $\hat\eta(X)$ computed from all $n$ observations and then proceeds as if that estimate were the truth. The correction to be made at the outset concerns which comparison is meaningful. Against a fixed-prior Bayes estimator whose prior is correct, empirical Bayes must lose, and the amount it loses is computed below. Against the maximum likelihood estimator, empirical Bayes in the canonical normal model cannot lose at all — its risk is strictly smaller for every value of the parameters, a theorem rather than a tendency. The suggestion that estimating hyperparameters can make an estimator worse than maximum likelihood is, in this setting, false, and the reason it is false is instructive. The model, and the estimator it produces Take the model on which every result below is exact. Each observation is normally distributed about its own parameter with known variance, and the parameters are themselves normal about a common centre: $$X_i \mid \theta_i \sim N(\theta_i, \sigma^2), \qquad \theta_i \sim N(\mu, \tau^2), \qquad i = 1,\dots,n,$$ with the $\theta_i$ independent. Integrating out $\theta_i$ gives the marginal distribution of what is actually seen, $X_i \sim N(\mu, \sigma^2+\tau^2)$, which is the fact that makes hyperparameter estimation possible: the observations carry information about $\tau^2$ even though no $\theta_i$ is observed. Conjugacy of the normal family gives the posterior in closed form. Writing $$B \;=\; \frac{\sigma^2}{\sigma^2+\tau^2},$$ the posterior distribution of $\theta_i$ given $X_i$ is normal with mean $(1-B)X_i + B\mu$ and variance $(1-B)\sigma^2$. The quantity $B$ is the shrinkage factor , and reading it settles most of the intuition. It lies between $0$ and $1$; it is close to $1$ when the observation noise $\sigma^2$ dominates the spread $\tau^2$ of the true parameters, in which case each $X_i$ is mostly noise and the estimate is pulled almost all the way to the common centre $\mu$; it is close to $0$ when the parameters are widely dispersed relative to the noise, in which case each observation is informative about its own parameter and is left nearly alone. The posterior variance $(1-B)\sigma^2$ is smaller than $\sigma^2$ by exactly the same factor, which is the sense in which the prior buys precision. Only $B$ needs to be known, not $\sigma^2$ and $\tau^2$ separately. Take $\mu = 0$ for simplicity, which loses nothing essential. The Bayes estimator is then $\hat\theta_i = (1-B)X_i$, and the whole question becomes what happens when $B$ is estimated rather than given. Estimating the shrinkage factor, and where the $n-2$ comes from Since $X_i \sim N(0, \sigma^2+\tau^2)$ marginally and the observations are independent, the sum of squares $\|X\|^2 = \sum_i X_i^2$ has the distribution of $(\sigma^2+\tau^2)$ times a chi-squared variable with $n$ degrees of freedom. That sum is therefore the natural source of information about $\sigma^2+\tau^2$, and hence about $B$. The obvious estimator $\sigma^2 n/\|X\|^2$ is biased, because the expectation of a reciprocal is not the reciprocal of an expectation. The correction is exact and explains a constant that otherwise looks arbitrary. For a chi-squared variable $W$ with $n$ degrees of freedom, $E[1/W] = 1/(n-2)$, provided $n \ge 3$; the expectation is infinite for $n = 1$ and $n = 2$, because the density of $W$ near zero does not decay fast enough for $1/W$ to be integrable. Hence $$E\!\left[\frac{(n-2)\,\sigma^2}{\|X\|^2}\right] \;=\; \frac{(n-2)\,\sigma^2}{\sigma^2+\tau^2}\cdot\frac{1}{n-2} \;=\; B .$$ The numerator is $n-2$ rather than $n$ precisely because that is the constant which makes the estimator unbiased for $B$, and the restriction to $n \ge 3$ is not a technicality but the condition under which the quantity being corrected is finite at all. Substituting this $\hat B$ into the Bayes estimator gives $$\hat\theta^{\mathrm{JS}} \;=\; \left(1 - \frac{(n-2)\,\sigma^2}{\|X\|^2}\right) X,$$ which is the James-Stein estimator of James and Stein (1961). Efron and Morris (1973) showed that it is exactly the empirical Bayes rule for this model, which is the fact that makes the general question about empirical Bayes answerable in closed form here. The exact cost of estimating the prior Three estimators can now be compared on Bayes risk — the total expected squared error $E\|\hat\theta - \theta\|^2$, averaged over both the noise and the prior. Averaging over the prior is the fair standard here because it is the standard under which the fixed-prior Bayes rule is optimal by construction; any comparison it wins under a different criterion would prove not