Unpacking the Rigour of Statistical Independence — Epoche C1
Research Note: On the Precise Definition of Statistical Independence A common misconception, particularly among those new to probability and statistics, is the conflation of statistical independence with the mere absence of correlation. While it is true that independent random variables are always uncorrelated, the converse does not hold. This note aims to clarify why statistical independence represents a far more profound and restrictive condition than a simple lack of linear association, delving into its formal definition, the reasons behind each step of the standard derivations, and its informational implications. What Independence Asks of a Pair of Variables Begin with the objects themselves. A random variable is a rule that attaches a number to each possible outcome of a random experiment: the number of heads in ten coin tosses, the height of a person drawn from a population. An event is a set of outcomes to which a probability can be assigned — in the technical vocabulary, a measurable set ; the qualifier exists because in continuum settings one can construct pathological sets to which no consistent probability can be attached, so the theory restricts attention to a family of well-behaved sets. Nothing below turns on that subtlety; the reader may safely read 'measurable set' as 'any set you could actually ask a probability question about', such as an interval. At its core, statistical independence between two random variables, say $X$ and $Y$, means that knowledge of the outcome of one provides no information whatsoever about the outcome of the other. This is rigorously defined through the behaviour of their joint and marginal distributions — the joint distribution describing probabilities of statements about the pair $(X,Y)$ together, the marginals describing $X$ and $Y$ each on their own. Formally, $X$ and $Y$ are independent if, for all measurable sets $A$ and $B$, the joint probability factors into the product of the marginals: $$ P(X \in A, Y \in B) = P(X \in A)\, P(Y \in B). $$ The quantifier is the whole point. The definition does not ask that factorisation hold for one convenient pair of sets, or on average; it must hold for every question one could pose about $X$ simultaneously with every question about $Y$. That is infinitely many constraints at once, and it is why independence is so restrictive: a single pair of sets $A, B$ for which the equation fails is enough to destroy independence, as the worked example below will show concretely. The definition extends to the standard descriptive functions. For a discrete variable the relevant object is the probability mass function (PMF), the list of probabilities of individual values, and independence is equivalent to $P(X=x, Y=y) = P(X=x)P(Y=y)$ for all values $x, y$ — here single points already generate all events. For continuous variables one uses the probability density function (PDF) $f_X$, the function whose integral over an interval gives the probability of landing in that interval; independence is equivalent to the joint density splitting as $f_{X,Y}(x,y) = f_X(x)f_Y(y)$ for all $x, y$. Both statements follow from the event formulation because intervals (or points) generate all the events in question; the equivalence is proved in any measure-theoretic text, for instance Durrett (2010) or Billingsley (1995). Independence as Informational Invariance The crucial aspect of the definition lies in its implications for conditional probabilities. Recall the definition of conditioning: $P(B \mid A) = P(A \cap B)/P(A)$, the proportion of the probability of $A$ that is also in $B$ — 'given that $A$ happened, how likely is $B$?'. If $X$ and $Y$ are independent, then for any value $x$ with $P(X=x)$ positive, $$ P(Y=y \mid X=x) = \frac{P(X=x, Y=y)}{P(X=x)} = \frac{P(X=x)P(Y=y)}{P(X=x)} = P(Y=y), $$ where the middle step substitutes the factorisation and the last cancels the common factor. Observing $X=x$ leaves the distribution of $Y$ exactly where it was. This informational invariance is the true hallmark of independence, and it can be made quantitative. Information theory measures the total dependence between two variables by their mutual information $I(X;Y)$: the expected value of $\log\bigl(f_{X,Y}(x,y)/(f_X(x)f_Y(y))\bigr)$, i.e. how far, on average, the joint distribution departs from the product of its marginals. This quantity is the Kullback–Leibler divergence — a standard, always non-negative measure of discrepancy between two distributions — between the joint law and the product law. Its two defining properties, proved via Jensen's inequality (the fact that a concave function of an average is at least the average of the function, applied to the logarithm), are that $I(X;Y) \ge 0$ always, and that $I(X;Y) = 0$ precisely when $X$ and $Y$ are independent (Cover and Thomas, 2006). Mutual information therefore detects every form of dependence, linear or not. Correlation, as we now see, does not. What Correlation Actually Measures Three ingredients need their plain meanings. The expectation $E[X]$ is the probability-weighted average of $X$. The covariance , $\mathrm{Cov}(X,Y) = E[(X - E[X])(Y - E[Y])]$, averages the product of the two variables' deviations from their means: it is positive when the variables tend to be above (or below) their means together, negative when one tends to be high while the other is low. Expanding the product and using the linearity of expectation gives the equivalent working form $\mathrm{Cov}(X,Y) = E[XY] - E[X]E[Y]$. Finally, the standard deviation $\sigma_X$ is the square root of the variance $E[(X-E[X])^2]$, the natural scale of $X$'s fluctuations. The Pearson product-moment correlation coefficient standardises the covariance by these scales: $$ \rho_{X,Y} = \frac{\mathrm{Cov}(X,Y)}{\sigma_X \sigma_Y}. $$ The Cauchy–Schwarz inequality — which states that $|E[UV]| \le \sqrt{E[U^2]E[V^2]}$ for any two variables with finite second moments — forces $\rho_{X,Y}$ to lie between $-1$ and $1$, with the extre