INTERACTIVE COMPANION · METHODS · STATISTICS

The Paradox Constructor

A live model accompanying “Building a Dataset Where Every Group Trends Up and the Whole Trends Down”

Two machines below. The first lets you build Simpson's paradox with your hands: drag five groups around a plot and watch the line fitted to all of them disagree with every one of them. The second measures how often that disagreement turns up on its own, by simulating datasets in your browser and counting.

Both run the same arithmetic as analysis/simpsons-paradox.py, and the page opens on the settings that produce the article's headline numbers.

Model 1. Build one by hand

Five groups sit on the plot. Each one holds points arranged on a line of slope $b_w$, which starts at $+1$, and each group's filled marker is its mean. Drag a marker and the group moves with it. The single line fitted to all the points is drawn heavy and dark.

The identity underneath is the one the article turns on. Write $T_w$ for the spread of $x$ inside groups, $T_b$ for the spread of the group means, and $w = T_w/(T_w+T_b)$. Then

$$b_{\text{pooled}} = w\,b_w + (1-w)\,b_b,$$

with $b_b$ the slope through the group means weighted by group size. The readout prints both the fitted pooled slope and the value this identity predicts, so you can watch them stay equal to the last decimal while everything else moves.

Within-group slope +1.00 Every group carries this slope. Drag it through zero and watch the warning line invert: the paradox needs the two slopes to disagree in sign, and it does not care which of them is positive. Group in hand 1 Picks which group the size slider acts on. The selected group's marker is outlined. You can also drag any marker directly, or focus one with the keyboard and nudge it with the arrow keys. Size of that group 15 points Push one group to 120 and the rest stay at 15. The pooled line swings toward that group's own slope, because between-group leverage is a product $n_1 n_2 / (n_1 + n_2)$ and products of a fixed sum are largest at balance. This is the §7 result you can feel with your hand. Noise inside groups 0.00 Zero gives the exact construction, which is obviously artificial. Push it to 0.8 and the five groups become five modest positive findings while the pooled line stays emphatically negative. That version is the one that would get published. Preset “All groups stacked” is the control: put every group at the same $x$ and $T_b$ becomes zero, so $w = 1$ and the pooled slope equals the within-group slope exactly. No reversal is possible from there, and the readout says so.

pooled slope: – identity says: – w: – b_between: – N: –

–

Model 2. Count how often it happens by accident

Now stop building and start counting. Each trial generates a fresh dataset: $G$ groups totalling 240 points, every group carrying a true slope of $+1$, residual standard deviation 1. A group-level lurking variable $z_g$ is drawn standard normal, shifts each group's mean in $x$ by $c z_g$ and its level in $y$ by $c\theta z_g$, and the loading $\theta$ is redrawn from $N(0,\tau^2)$ every trial. So the confounder pushes with the real effect half the time and against it half the time, which is what “by accident” has to mean.

A trial counts as a reversal when every within-group slope has one sign and the pooled slope has the other. The gated count is stricter: every one of those slopes must also clear a two-sided $t$ test at $\alpha = 0.05$, so it is the rate at which a careful analyst would end up holding two confident findings pointing opposite ways.

Confounder strength c 1.00 The between-group spread of $x$, measured in within-group spreads. Leave it at 1 and the estimate should settle near 0.0970, the article's value from 20,000 trials. Slide it to 0 and the count stops dead at zero, which it must, because with no between-group displacement the pooled slope equals the within-group slope identically. Group-size ratio 1:1 Largest group over smallest, with the total held at 240. Push it to 30 and the reversal rate falls by half, from about 0.097 to about 0.049. Unequal groups are a protection here, not a hazard, which is the result in §7 we would most like somebody to break. Number of groups G 4 More groups means less of the spread in $x$ sits inside them. At $G = 2$ the article measures 0.056 and at $G = 12$ it measures 0.133. Finer stratification of the same data raises the exposure. Confounder loading SD τ 1.00 How hard the lurking variable pulls on $y$ relative to its pull on $x$. This is the softest joint in the whole study. At $\tau = 0.5$ the rate is 0.029 and at $\tau = 2$ it is 0.193, so a reader who thinks confounders in their field are half our size should read the smaller number.

trials: 0 reversal rate: – SE: – gated rate: – mean w: –

–

Three things to check by hand

A model you cannot check is a cartoon. Three of the numbers on this page can be verified with arithmetic, and if any of them disagrees with what you see, the page is broken and we would like to know.

Model 1, opening settings: pooled slope −1.000000, within-group slopes +1.000000
the sum behind it: T_w = 85.714286, T_b = 600, w = 1/8, b_b = −9/7
and so: (1/8)(+1) + (7/8)(−9/7) = 0.125 − 1.125 = −1 exactly
Model 1, group 1 mean: y = −(9/7)(−4) = 36/7 = 5.142857
Model 1, stacked preset: T_b = 0, w = 1, so the pooled slope equals the size-weighted average of the within-group slopes exactly
Model 2 at c = 0: reversal rate exactly 0, however long you run it
Model 2 at the defaults: converges on 0.09695 ± 0.00209, the article's 20,000-trial value
Model 2 at 30:1 sizes: converges on 0.04855, about half the balanced rate

The last two are the only ones with sampling error in them. This page runs fewer trials than the article does, so expect the estimate to wander by a few thousandths before it settles. The band drawn around the running curve is one standard error and the dashed line is the article's answer, which carries a standard error of its own. The readout compares the two on the standard error of their difference rather than on this page's alone, because comparing an estimate to another estimate as though the second were exact is a way of manufacturing disagreements. We ran 40,000 trials of this code outside the browser and got 0.09405, which is 1.14 such standard errors from the article's 0.09695. Past three would mean one of the two implementations has a bug in it.

What this page cannot show you

It cannot tell you which slope is the right one to report. Nothing can, from the data alone. The paradox is a fact about weighted averages and carries no information about what caused what, so deciding between the within-group answer and the pooled answer needs a causal model that no amount of dragging will supply.

It also flatters the world in three specific ways. Every group here shares one slope, which real groups never do, and unequal slopes make reversals easier rather than harder. The errors are Gaussian, and heavier tails would raise every frequency on the second panel. And the groups are labelled: you can see them. The case that should actually worry a reader is the one where the confounder was never recorded, where the arithmetic is identical and the plot is a single undifferentiated cloud with one line through it.