INTERACTIVE COMPANION · EXPLAINER · STATISTICS

The Inference Bench

A live model accompanying “What a P Value Actually Means”

Two benches. The first runs simulated experiments in front of you, one every few milliseconds, dropping each resulting p value into a histogram so you can watch the distribution build. With the true effect set to zero the bars come out flat, which is the property that defines the p value and is much more convincing when you see it happen than when somebody tells you. Push the effect up and the whole pile slides into the leftmost bin.

The second bench asks the harder question. Given a field where some fraction of tested hypotheses are true, and studies of a given power, what share of the significant results are real? It draws the tree of outcomes to scale, so the two boxes you are comparing are the actual sizes of the two groups.

Bench 1. Watch the distribution build

Every tick, the model draws two groups of $n$ observations from normal populations separated by $d$ standard deviations, runs a two-sided t test, and files the p value. Nothing is precomputed. The bars you see are the experiments that have actually run since you last changed a slider.

Start with the defaults, which are the article's $d = 0$ case, and let it run to a few thousand experiments. The bars settle at 0.05 each because the p value under a true null is uniform. The dark leftmost bar is everything that would be published as a discovery, and there is nothing to discover.

True effect size d 0.00 SD Zero means the two populations are identical and the null hypothesis is exactly true. Let it run flat for a while, then drag to 0.50 and watch the leftmost bar climb to about half. At 0.80 it takes nearly everything. The distribution never becomes a spike, which is the point of the article's Figure 3. Observations per group n 32 Set d to 0.20, a small but real effect, then walk n from 10 up to 200. The same effect goes from nearly invisible to nearly certain. This is why a p value on its own cannot tell you how big anything is. Significance threshold α 0.050 Moves the dark line. With d at zero the fraction of runs below the line should track the threshold exactly, whatever you set it to, because the distribution is flat. That equality is the definition, and you can watch it hold. Speed The fast setting reaches ten thousand experiments in a few seconds, which is where the bars stop wobbling.

experiments run: 0 below α: 0 observed rate: – predicted power: –

–

Bench 2. What fraction of significant findings are real?

Out of 1,000 studies in a field, $R$ of them per unit test a true hypothesis. True hypotheses are caught with probability equal to the power. False ones are called significant with probability $\alpha$. So:

$$\mathrm{PPV} \;=\; \frac{R \times \text{power}}{R \times \text{power} \;+\; (1-R)\,\alpha}$$

The tree below is drawn to scale. The width of each branch is the number of studies flowing down it, and the two filled boxes at the bottom are the true and false positives whose ratio you are being asked to judge. The defaults are the article's headline case.

Prior R: share of tested hypotheses that are true 10% Drag down to 1%, which is roughly a screening programme running through a long candidate list, and watch the true-positive box shrink to a sliver. Drag up to 50%, a field testing well-motivated hypotheses from prior work, and the same alpha suddenly buys you a reliable literature. The prior does more work than anything else on this page. Power: chance of catching a true effect 50% Button and colleagues put the median in neuroscience near 21%. Set it there with the prior at 10% and read the bottom line. Then take it to 95% and notice how much less it helps than you expect, because power cannot touch the false positives at all. Threshold α 0.050 This is the only lever that shrinks the false-positive box. Move it to 0.005 and the share wrong in the headline case drops from 47.4% to 8.3%. The honest version of that move also costs you power, which this model deliberately does not do for you, so treat it as an upper bound on the benefit. Multiplicity: independent tests per study Running several tests and reporting the best one inflates the real false-positive rate to $1-(1-\alpha)^k$. At k = 20 and α = 0.05 that is 0.64, and the tree redraws with the honest number.

true positives: – false positives: – significant findings that are real: – that are false: –

–

Club headline: prior 10%, power 50%, α 0.05
Per 1,000 studies: 50 true positives, 45 false positives
PPV: 52.6% real, 47.4% false
Simulated check: 0.5279 over 400,000 studies, against 0.5263 analytic
Break-even prior: 9.1% at power 0.50: below that, most significant findings are false

Three things worth trying

Set Bench 1 to $d = 0$ and α to 0.01. Let it run. The observed rate walks toward 0.010, not 0.05, because the flat distribution hands you whatever tail fraction you ask for. Nothing about the data changed; you moved a line.

Set Bench 1 to $d = 0.5$ and $n = 32$, the article's case, and watch the observed rate settle near 0.50. Now imagine two labs both drawing from that histogram. The chance they land on opposite sides of the threshold is $2 \times 0.5 \times 0.5$, which is one half. That is the dance of the p values, and Bench 1 is drawing the distribution both labs are sampling from.

On Bench 2, put the prior at 1% and the power at 20%, then switch multiplicity to 20 undeclared outcomes. The share of significant findings that are real falls below one percent. Every study in that scenario is honest in the sense that nobody lied. The arithmetic still eats the literature.