A Statistical Paradox

How Wheat Prices
Help Predict Baseball Averages

In 1956, Charles Stein proved something that still confuses statisticians: unrelated data improves estimates.

Here's an absurd claim: If you want to estimate a baseball player's true batting average, you should look at wheat prices.

Not metaphorically. Not as a sanity check. Actually use them in your calculation. Your estimate will be more accurate.

This isn't a trick. It's called Stein's Paradox, and it broke statistics in 1956. The proof is airtight. The math is correct. And yet it feels deeply, fundamentally wrong.

Let me show you why it works.

But first, we need to understand what "better estimate" even means.

PART I

The Estimation Game

Before we get to the paradox, let's play a game. I'll think of a number. You'll see a noisy observation of it. Your job is to guess the true value.

I'll generate a random true value from a normal distribution. You'll see it plus some noise. Try to guess the original.

When you have one thing to estimate, your best strategy is obvious: just use the observation. It's unbiased. Any shrinkage toward zero or any other value would add bias.

Formally, the MLE simply sets $\hat{\theta}_i^{\text{MLE}} = X_i$ -- your estimate equals your observation. It maximizes the likelihood $\mathcal{L}(\theta \mid X) = \prod_{i} \frac{1}{\sqrt{2\pi}\sigma} \exp\!\left(-\frac{(X_i - \theta_i)^2}{2\sigma^2}\right)$ .

For two things? Same answer. Estimate each one with its observation.

But for three or more?

PART II

The Stein Shock

In 1956, Charles Stein proved that for three or more parameters, the MLE is inadmissible.

That word has a precise meaning: there exists another estimator that is always at least as good, and sometimes strictly better, no matter what the true values are.

The James-Stein estimator: shrink all your estimates toward their common mean.

SIMULATION: MLE vs JAMES-STEIN

Dimensions:

Click "Run 100 simulations" to see the paradox in action

Run the simulation with 1 or 2 dimensions: MLE wins about half the time. There's no improvement.

Now try 3 or more. Watch James-Stein win consistently. The more dimensions, the bigger the advantage.

This is the paradox. At exactly n=3, something fundamental changes.

RISK vs DIMENSIONS

The gap between MLE and James-Stein risk grows with dimensionality. At p=3 the improvement begins; by p=20 it's dramatic.

ERROR COMPARISON SCATTER

p =

Each dot is one simulation. Points below the diagonal mean James-Stein had lower error. Nearly all points fall below — the paradox in action.

RISK REDUCTION BY DIMENSION (Monte Carlo, n=1000)

No improvement for p=1,2 (greyed out). At p=3 the paradox begins. By p=50 James-Stein reduces risk by ~80%.

3D RISK RATIO SURFACE — R(James-Stein) / R(MLE)

Blue regions show where James-Stein dramatically outperforms MLE. The improvement is strongest when p is large and ||θ||² is small (parameters near the shrinkage target).

The Mathematics of James-Stein

A step-by-step derivation from observation model to risk proof

1

The Setup

We observe $p$ independent measurements, each corrupted by Gaussian noise:

X_i = \theta_i + \varepsilon_i, \quad \varepsilon_i \overset{\text{iid}}{\sim} \mathcal{N}(0, \sigma^2), \quad i = 1, \dots, p

In vector notation: $X \sim \mathcal{N}(\theta, \sigma^2 I_p)$ . We want to estimate $\theta = (\theta_1, \dots, \theta_p)$ and we measure quality by total squared error:

L(\delta, \theta) = \|\delta - \theta\|^2 = \sum_{i=1}^{p} (\delta_i - \theta_i)^2

The risk of an estimator $\delta$ is its expected loss:

R(\delta, \theta) = E_\theta\!\left[\|\delta(X) - \theta\|^2\right]

2

The Maximum Likelihood Estimator

The MLE is the obvious estimator: just use the observations directly.

\delta^{\text{MLE}} = X

Its risk is straightforward to compute:

R(\delta^{\text{MLE}}, \theta) = E\!\left[\|X - \theta\|^2\right] = E\!\left[\sum_{i=1}^p \varepsilon_i^2\right] = p\sigma^2

This risk is constant -- it does not depend on $\theta$ at all. It grows linearly with dimension $p$ . For 10 parameters, the expected total squared error is $10\sigma^2$ .

For $p = 1$ or $p = 2$ , the MLE is admissible -- no estimator can uniformly beat it. This was proven by Hodges & Lehmann (1950). The surprise is what happens at $p = 3$ .

3

The James-Stein Estimator

James and Stein (1961) proposed shrinking observations toward the origin:

\delta^{\text{JS}} = \left(1 - \frac{(p-2)\sigma^2}{\|X\|^2}\right) X

The shrinkage factor $B = \frac{(p-2)\sigma^2}{\|X\|^2}$ determines how much we pull toward zero. When $\|X\|^2$ is large (data far from origin), we shrink less. When data clusters near zero, we shrink more.

More generally, we can shrink toward any point $\mu$ (typically the grand mean $\bar{X}$ ):

\delta^{\text{JS}}_\mu = \bar{X}\mathbf{1} + \left(1 - \frac{(p-2)\sigma^2}{\|X - \bar{X}\mathbf{1}\|^2}\right)(X - \bar{X}\mathbf{1})

The positive-part variant prevents overshooting (shrinking past the target):

\delta^{\text{JS+}} = \bar{X}\mathbf{1} + \max\!\left(0,\; 1 - \frac{(p-2)\sigma^2}{\|X - \bar{X}\mathbf{1}\|^2}\right)(X - \bar{X}\mathbf{1})

4

The Risk Proof (Stein's Unbiased Risk Estimate)

The key result: James-Stein has strictly lower risk than MLE for all $\theta$ when $p \geq 3$ .

R(\delta^{\text{JS}}, \theta) = p\sigma^2 - (p-2)^2 \sigma^4 \, E_\theta\!\left[\frac{1}{\|X\|^2}\right]

Since $\|X\|^2 > 0$ almost surely (and $E[1/\|X\|^2]$ is finite when $p \geq 3$ ), the second term is strictly positive. Therefore:

R(\delta^{\text{JS}}, \theta) < p\sigma^2 = R(\delta^{\text{MLE}}, \theta) \quad \text{for all } \theta

Proof sketch using Stein's Lemma: If $X \sim \mathcal{N}(\theta, \sigma^2 I)$ and $g: \mathbb{R}^p \to \mathbb{R}^p$ is weakly differentiable, then:

E\!\left[(X - \theta)^\top g(X)\right] = \sigma^2 E\!\left[\nabla \cdot g(X)\right]

Applying this with $g(X) = -\frac{(p-2)\sigma^2}{\|X\|^2} X$ yields the risk formula above. The divergence calculation produces the $(p-2)^2 \sigma^4 / \|X\|^2$ term.

5

Why p = 3 is the Threshold

The factor $(p-2)$ in the estimator is the key. Consider what happens at each dimension:

p=1

-1

Negative -- shrinkage would increase risk

p=2

0

Zero -- no shrinkage, reduces to MLE

p=3

1

First positive value -- shrinkage begins to help

p=p

p-2

Risk reduction grows as (p-2)^2 / p

Deeper reason: $E[1/\|X\|^2]$ is finite only when the chi-squared distribution $\|X\|^2 / \sigma^2 \sim \chi^2_p(\|\theta\|^2/\sigma^2)$ has enough degrees of freedom. The expectation $E[1/\chi^2_p]$ exists only for $p \geq 3$ (requires $p - 2 > 0$ degrees of freedom after the $1/x$ transformation).

As $p \to \infty$ , the fractional risk improvement approaches:

\frac{R(\delta^{\text{MLE}}) - R(\delta^{\text{JS}})}{R(\delta^{\text{MLE}})} \approx \frac{(p-2)^2}{p \cdot (\|\theta\|^2/\sigma^2 + p)} \to 1 - \frac{\|\theta\|^2/\sigma^2}{p}

For large $p$ with bounded $\|\theta\|^2$ , shrinkage captures almost all of the MLE's excess risk.

6

Connection to Empirical Bayes

The James-Stein estimator is the empirical Bayes estimator under a specific prior. Suppose:

\theta_i \overset{\text{iid}}{\sim} \mathcal{N}(0, A), \quad X_i \mid \theta_i \sim \mathcal{N}(\theta_i, \sigma^2)

The Bayes estimator (posterior mean) is:

E[\theta_i \mid X_i] = \frac{A}{A + \sigma^2} X_i = \left(1 - \frac{\sigma^2}{A + \sigma^2}\right) X_i

We don't know $A$ , but we can estimate $A + \sigma^2 \approx \|X\|^2 / p$ . Plugging in:

\frac{\sigma^2}{A + \sigma^2} \approx \frac{p\sigma^2}{\|X\|^2}

Adjusting for estimation uncertainty gives the $(p-2)$ factor instead of $p$ , yielding exactly the James-Stein estimator. The shrinkage is an estimated Bayes rule -- it learns the prior from the data itself.

PART III

The Geometric Intuition

Here's the intuition. Think about what happens when you observe noisy data.

In one dimension, noise pushes your observation left or right with equal probability. On average, you're at the right distance from zero.

In high dimensions, something weird happens. Almost all the volume of a high-dimensional sphere is near the surface, not the center. This is the concentration of measure phenomenon:

P\!\left(\left|\frac{\|X\|}{\sqrt{p}} - 1\right| > t\right) \leq 2e^{-pt^2/8}

The norm of a high-dimensional Gaussian concentrates tightly around $\sqrt{p}$ .

3D MONTE CARLO — TRUTH vs MLE vs JAMES-STEIN

Green = true values, Red = MLE estimates (raw observations), Purple = James-Stein (shrunk toward mean). Notice how purple dots are systematically closer to green dots — shorter dashed lines = less error.

This means in high dimensions, random noise almost always pushes observations outward, away from the truth.

The MLE just uses these inflated observations directly. It systematically overshoots.

James-Stein corrects for this by shrinking toward the center. It's not magic -- it's geometry.

3D LOSS LANDSCAPE

p =

The loss valley runs along the optimal shrinkage ridge (orange line). MLE sits at c=1 (right edge) — always higher risk than the valley floor. The valley is deeper (more improvement) for larger p.

3D MSE DECOMPOSITION — BIAS² + VARIANCE

Orange solid = bias² (grows as shrinkage c decreases from 1). Purple wireframe = total MSE (bias² + variance). The gap between them is the variance contribution. The optimal c minimizes the purple surface — always below MLE at c=1.

BIAS-VARIANCE TRADEOFF

Shrinkage introduces bias but reduces variance. The optimal point (green) achieves lower total risk than the unbiased MLE (c=1). James-Stein finds this sweet spot automatically.

SHRINKAGE FACTOR DISTRIBUTION

The shrinkage factor B = max(0, 1 - (p-2)/||X - X̅||²) varies per sample. Observations far from the mean get shrunk less; clustered ones get shrunk more.

3D SHRINKAGE VISUALIZATION

Dimensions:

MLE distance to truth

2.323

JS distance to truth

1.917

Shrinkage factor

0.926

How much to shrink? (0 = MLE, 1 = all the way to mean)

No shrinkage (MLE)50%Full shrinkage

The optimal shrinkage is $B = \frac{(p-2)\sigma^2}{\|X\|^2}$ , which James-Stein calculates automatically from the data. At your setting of 50%, each estimate becomes $\hat{\theta}_i = \bar{X} + 0.50(X_i - \bar{X})$ .

PART IV

Examples and Connections

Worked Examples

See James-Stein shrinkage in action on real-world-style data

Baseball Batting Averages (Efron-Morris 1977)

18 MLB players' batting averages estimated from their first 45 at-bats of the season, compared to their final season averages. This is the classic demonstration of Stein's paradox.

Grand mean: 0.262 | Shrinkage factor

B

: 0.234

Player	Observed	JS Estimate	True Value	MLE Error	JS Error
Roberto Clemente	0.400	0.294	0.346	0.054	0.052
Frank Robinson	0.378	0.289	0.298	0.080	0.009
Frank Howard	0.356	0.284	0.276	0.080	0.008
Jay Johnstone	0.333	0.278	0.222	0.111	0.056
Ken Berry	0.311	0.273	0.273	0.038	0.000
Jim Spencer	0.311	0.273	0.270	0.041	0.003
Don Kessinger	0.289	0.268	0.263	0.026	0.005
Luis Alvarado	0.267	0.263	0.210	0.057	0.053
Ron Santo	0.244	0.258	0.269	0.025	0.011
Ron Swoboda	0.244	0.258	0.230	0.014	0.028
Rico Petrocelli	0.222	0.252	0.264	0.042	0.012
Ellie Rodriguez	0.222	0.252	0.226	0.004	0.026
George Scott	0.222	0.252	0.303	0.081	0.051
Del Unser	0.200	0.247	0.264	0.064	0.017
Billy Williams	0.200	0.247	0.256	0.056	0.009
Bert Campaneris	0.178	0.242	0.286	0.108	0.044
Thurman Munson	0.178	0.242	0.316	0.138	0.074
Max Alvis	0.156	0.237	0.200	0.044	0.037

MLE MSE

0.0047

JS MSE

0.0012

Improvement

73.6%

Hospital Mortality Rates

Estimating true mortality rates across 10 hospitals from limited annual data (~500 patients each). Shrinkage toward the grand mean reduces estimation error, which matters for fair quality comparisons.

Grand mean: 0.062 | Shrinkage factor

B

: 0.749

Hospital	Observed	JS Estimate	True Value	MLE Error	JS Error
City General	0.082	0.077	0.065	0.017	0.012
St. Mary's	0.045	0.049	0.052	0.007	0.003
Regional Med	0.071	0.069	0.060	0.011	0.009
University Hosp	0.093	0.085	0.070	0.023	0.015
Community Care	0.038	0.044	0.048	0.010	0.004
Memorial	0.067	0.066	0.058	0.009	0.008
Mercy Hospital	0.055	0.057	0.055	0.000	0.002
Veterans Med	0.078	0.074	0.062	0.016	0.012
Children's	0.031	0.039	0.040	0.009	0.001
Sacred Heart	0.060	0.061	0.057	0.003	0.004

MLE MSE

0.0002

JS MSE

0.0001

Improvement

54.0%

Student Test Scores Across Subjects

5 students, scores on Math/Reading/Science. A single test is a noisy measure of true ability. Shrinking toward the grand mean of all scores helps estimate each student's per-subject ability.

Grand mean: 78.133 | Shrinkage factor

B

: 0.563

Student - Subject	Observed	JS Estimate	True Value	MLE Error	JS Error
Alice - Math	92.000	85.946	85.000	7.000	0.946
Alice - Reading	78.000	78.058	80.000	2.000	1.942
Alice - Science	88.000	83.692	82.000	6.000	1.692
Bob - Math	65.000	70.734	72.000	7.000	1.266
Bob - Reading	81.000	79.748	75.000	6.000	4.748
Bob - Science	70.000	73.551	73.000	3.000	0.551
Carol - Math	95.000	87.636	88.000	7.000	0.364
Carol - Reading	72.000	74.678	78.000	6.000	3.322
Carol - Science	84.000	81.439	80.000	4.000	1.439
Dan - Math	58.000	66.790	65.000	7.000	1.790
Dan - Reading	69.000	72.987	68.000	1.000	4.987
Dan - Science	62.000	69.043	66.000	4.000	3.043
Eve - Math	88.000	83.692	82.000	6.000	1.692
Eve - Reading	91.000	85.383	85.000	6.000	0.383
Eve - Science	79.000	78.622	83.000	4.000	4.378

MLE MSE

29.2000

JS MSE

6.9831

Improvement

76.1%

Stein's paradox isn't just a statistical curiosity. It's the theoretical foundation for modern machine learning.

Regularization

L2 regularization (Ridge regression) is James-Stein shrinkage. Penalizing large weights is the same as shrinking toward zero.

Bayesian Priors

Even a "wrong" prior helps. Shrinking toward any value beats MLE in high dimensions. The prior is doing the same work.

Empirical Bayes

Use data to estimate the prior, then apply it. This is exactly what James-Stein does -- estimate shrinkage from the observations.

Neural Network Weights

Weight decay, dropout, batch normalization -- all forms of shrinkage. Modern deep learning is Stein's paradox at scale.

So yes, wheat prices really do help predict batting averages.

Not because they're related. Because in high dimensions, shrinking toward any common value beats treating each estimate independently.

The "unrelated" data provides the shrinkage target.

Stein's paradox tells us: in a complex world with many parameters, borrowing strength from everywhere beats going it alone.

PART V

How Wheat Prices
Help Predict Baseball Averages

The Estimation Game

The Stein Shock

The Mathematics of James-Stein

The Setup

The Maximum Likelihood Estimator

The James-Stein Estimator

The Risk Proof (Stein's Unbiased Risk Estimate)

Why p = 3 is the Threshold

Connection to Empirical Bayes

The Geometric Intuition

Examples and Connections

Worked Examples

Baseball Batting Averages (Efron-Morris 1977)

Hospital Mortality Rates

Student Test Scores Across Subjects

Further Reading

Want More Explainers Like This?