What a P Value Actually Means, and the Four Things People Think It Means Instead

Say the Sentence Out Loud

Read the next line twice. Out loud, if the room allows.

A p value is the probability of observing data at least as extreme as the data you observed, if the null hypothesis were exactly true.

The null hypothesis is the boring version of the world: the drug does nothing, the two groups came from the same population, the coefficient is zero, the coin is fair. You do not believe it. Nobody really does. You assume it anyway, temporarily and in bad faith, because the calculation needs somewhere to stand. Now your data are a random draw from a known distribution, and you can ask where in that distribution your particular result landed. Far out in the tail? Then the boring world finds your data surprising. The p value says how far out, as an area under a curve.

Now look at what the sentence leaves out. No probability that the null hypothesis is true. No size of any effect. No verdict on whether the result matters, and no promise that the lab down the corridor, running your study again next week, will land anywhere near where you did. The thing runs one way only: given the hypothesis, how weird are the data. Read it backwards and you have invented every mistake in this article.

We wrote this down because the club kept explaining the same four things at the table, month after month, usually to somebody who had already sat through it twice. Once, properly, on paper. Then we can point.

The Coffee Machine, and Why the Definition Is Backwards From What You Want

Suppose the club decides the coffee machine in the science corridor makes people faster at the Wednesday problem set. We grab 32 members. Half get coffee, half get hot water that looks like coffee. Everybody works the same twenty questions. Coffee group mean: 14.2 minutes. Water group: 15.9 minutes. Difference of 1.7 minutes, coffee wins, \(p = 0.031\).

What did we just learn?

Most readers hear this: coffee has a 3.1% chance of doing nothing. That would be a genuinely useful number to own. Nobody computed it. Wanting a number has never once summoned it. Here is what we actually computed. Assume coffee does nothing at all. Assume our sampling and our test are what we said they were. A gap of 1.7 minutes or larger then turns up in 3.1% of studies like ours, purely from which members landed in which group.

The two sentences sound alike in English, which is why people who would never confuse anything else trade one for the other. They relate the way "the probability of clouds given rain" relates to "the probability of rain given clouds." Rain all but guarantees clouds. Clouds guarantee you almost nothing. The swap has a name, the transposed conditional, and it fathers three of the four misreadings below [6, 7]. Learn the name. You will start seeing it everywhere.

Look at the picture before you argue with the sentence. Figure 1 draws the definition for this exact experiment, and §1 stops sounding like a technicality the moment you have seen the area it names.

The Property That Actually Defines It

Definitions given in words slide around under pressure. So here is the version you can put on a computer and check in ten seconds, which is the version the club trusts and goes back to whenever somebody starts hedging.

Under a true null hypothesis, p values are uniformly distributed between 0 and 1, meaning every value turns up as often as every other. A p of 0.02 is exactly as probable as a p of 0.72. And 5% of the time the number drops below 0.05, purely because random numbers behave like that. Stop calling it a side effect. Somebody specified it. The p value gets built by feeding your test statistic through the null distribution's own tail area, and that transformation flattens any distribution into a uniform one. Murdoch, Tsai and Adcock argue that simulation is how the thing should be taught in the first place, because p is a random variable, and students who never see its distribution never quite believe it [2].

We ran it. Two hundred thousand simulated two-group experiments, 32 observations per group, drawn from populations identical by construction, each one analysed with an ordinary two-sided t test. The code sits linked at the top of this page, seed included.

Go back and look at the far-left bar of Figure 2. In a world holding no effects whatsoever, one experiment in twenty comes out with a publishable p value. The method has not failed you. That rate was agreed in advance, in writing, by everybody who ever wrote a methods section, as the price of having any rule at all. The rest of this article exists because of it.

Figure 3 runs the same code with a real effect underneath. Same test, same sample size. Only the true difference between the two populations changes.

Wrong Thing One: The Probability the Hypothesis Is True

Start with the big one. This misreading will not die, because it answers the question every person running the study actually wanted answered.

You want \(P(\text{hypothesis} \mid \text{data})\). The p value hands you \(P(\text{data or worse} \mid \text{hypothesis})\). Different quantities. To get from one to the other you need something the p value has never carried and cannot carry, namely how plausible the hypothesis looked before you ran the study, because Bayes' rule takes a prior as an input. Go looking for a prior inside a p value. You will not find one. So no posterior is on offer.

How wrong does it get? Berger and Sellke worked out the best case, the case most flattering to whoever is doing the transposing [10]. Pick the prior that flatters the evidence against the null as far as mathematics allows. Do your worst. A p value of exactly 0.05 still supports odds against the null of no better than about 2.5 to 1. Not 19 to 1, which is what "95% confident" sounds like it should buy you. Sellke, Bayarri and Berger later reduced the whole thing to a formula you can do on a phone: the strongest possible Bayes factor against the null is \(-e\,p\ln p\) [11]. At \(p = 0.05\) that comes to 0.407. Odds of 2.5 to 1, about the evidence in a coin landing heads twice.

Those are ceilings, not estimates. Whatever your real evidence is, it sits below every line above. A p value just under 0.05, the number that publishes papers and funds laboratories, amounts to a shrug.

Wrong Thing Two: A Small P Means a Big Effect

Invent two studies. Study A tests a new tutoring scheme on 24 students and finds an improvement of 11 marks, \(p = 0.04\); study B tests a phone app on 40,000 students and finds an improvement of 0.3 marks, \(p = 0.0000002\). Which effect is bigger?

Obviously the tutoring. Eleven marks against a third of a mark. And yet study B's p value is smaller by five orders of magnitude, so skim the two abstracts at the speed everybody reads abstracts and you come away certain that B found something enormous and A something marginal.

A p value jams effect size, sample size and noise together. You cannot unjam it. Roughly, the test statistic scales as the effect divided by the standard error, and the standard error shrinks as \(1/\sqrt{n}\), so:

So you can manufacture any p value you like out of any effect you like, by choosing how many people to recruit. Our simulation makes this concrete. An effect of \(d = 0.5\) at 32 per group reaches significance 50.4% of the time; the same \(d = 0.5\) at 400 per group reaches it 100.0% of the time, rounded to four decimal places, across 100,000 runs. Identical effect. Completely different p values. Nothing about the world changed. Only the number of people we asked.

So when somebody reports \(p < 0.001\) and stops there, they have told you exactly one thing, and it is this: the combination of effect size and sample size came out large. Which of the two? Not saying. Ask for the effect size and its interval, every time, including of the papers whose conclusions you like. A paper that reports neither has still told you something. The omission is the finding [5].

Wrong Thing Three, as Plain Arithmetic: 0.049 Against 0.051

Take the two-group comparison from §3. Thirty-two observations per group, sixty-two degrees of freedom, a two-sided t test. Both studies are honest. One reports \(p = 0.049\), the other \(p = 0.051\). Here is what separates them.

Eighteen thousandths of one standard error separate them. Nothing else does. One gets written up as a discovery, submitted, reviewed, cited, taught. The other gets a folder nobody opens again.

Nothing in the mathematics changes as you cross the threshold. Somebody drew a line on a continuous scale. Rosenthal and Gaito asked nineteen psychologists how much confidence they placed in results at various p levels, and the drop came out steepest between 0.05 and 0.10. A cliff in belief, standing where the evidence has none [12]. Gelman and Stern put the general case plainly: the difference between significant and not significant is not itself statistically significant [13].

Two more consequences follow from the same arithmetic. A study at \(p = 0.06\) and a study at \(p = 0.04\) are not in conflict. And test each of two groups against zero, announce that one cleared the line and the other did not, and you have compared nothing at all, an error Nieuwenhuis and colleagues found in half the neuroscience papers where it could occur [14].

Wrong Thing Four: Not Significant Means No Effect

Absence of evidence is not evidence of absence. Altman and Bland put that sentence into the BMJ in 1995 [15]. People have quoted it ever since, then gone off and done the thing anyway.

A p value above your threshold means one of two things and cannot tell you which: either no effect exists, or one exists and your study was too small to see it. Look back at Figure 3. At \(d = 0.5\), a real and respectable effect, our simulation returns \(p > 0.05\) in 49.6% of runs; cut the sample to 16 per group, leave the effect sitting exactly where it was at 0.5, and now 72.3% of studies miss it altogether. Every one of those misses gets written up as "no significant difference was found."

The word "found" is doing heavy lifting there. Nothing was found. The study looked with a torch too dim for the job and reported darkness.

Button and colleagues put median power across neuroscience near 21% [16]. Read that number next to the paragraph above, slowly. A field running at that power learns almost nothing from a non-significant result, because even real effects get missed four times out of five, so ask what effect sizes the study could actually have detected. If the confidence interval stretches from "harmful" through "nothing" to "miraculous", the honest summary is that nobody knows yet.

Base Rates, and How an Honest Field Can Still Be Mostly Wrong

Now the part that changes how you read journals. It took the club longest to swallow. One of us went home and re-derived it alone before she would accept it.

Suppose a field runs 1,000 studies. Some fraction of the hypotheses tested are actually true; call that fraction \(R\), the prior, and note that a true hypothesis gets detected with probability equal to the power. A false one gets called significant with probability \(\alpha\), the threshold, since that is what \(\alpha\) means, so out of 1,000 studies:

and the share of significant findings that are real, the positive predictive value, is the first over the sum of both. Ioannidis built his 2005 paper on exactly this arithmetic and gave it the title everybody remembers [3], and here is the club's version, with numbers picked to be unflattering rather than absurd: a field where 10% of tested hypotheses are true, running studies at 50% power.

Forty-seven percent of the published significant findings in that field are wrong. Nobody cheated. Every study used the right test, reported what it found, hit the alpha it promised and would survive an audit. The errors are structural. They come out of the base rate, and no single study can see its own base rate.

Algebra alone did not satisfy us, so we simulated it: 400,000 studies, each with a hypothesis drawn true with probability 0.10 and an effect size tuned to give exactly 50% power. Of the 37,996 that came out significant, 20,058 were real: simulated PPV 0.5279 against the analytic 0.5263, a difference of 0.0016, and Figure 4 turns the same arithmetic into a curve. Find your own field on it.

Two things jump out of that table. Power helps, and it helps less than you would hope: at a 1% prior, going from 20% power to 95% power takes you from 96.1% wrong to 83.9% wrong. The prior does the heavy lifting. A field testing well-motivated hypotheses grounded in earlier work stands somewhere entirely different from one screening ten thousand candidate genes, and the same p value means something different in each.

Which is why the proposal to move the default threshold to 0.005 has teeth [8]: hold our headline case fixed, change only alpha, and the false share drops from 47.4% to 8.3%. Lakens and eighty-seven co-authors object that one new default merely relocates the problem, and that researchers should justify their alpha against the specific costs of being wrong in their own field [9]. Both papers are worth your time. They agree entirely about the arithmetic.

Working Notes From the Club Table

Nadia: "If two labs run the same study on the same real effect, how often do they agree?"

Nobody knew. Somebody guessed 90%. Onto the board it went, with the arithmetic under it. Ninety seconds, start to finish: give each study power \(w\), keep the two independent, and they land on opposite sides of the threshold with probability \(2w(1-w)\).

Plug in \(w = 0.5\). Two times a half times a half. One half. Two competent studies of one genuine effect, both run correctly by people who know exactly what they are doing, will contradict each other half the time, and neither has done a single thing wrong.

Four seconds of silence. Then everybody talked at once.

200,000 pairs of studies, \(n = 32\) per group, true \(d = 0.5\). Simulated disagreement rate 0.5018. Analytic 0.5000. Correct, and always going to be. Cumming named this the dance of the p values [4]. Follow a first study at \(p = 0.05\) with a replication, and the second p value lands anywhere across a range roughly two orders of magnitude wide. Study size does not rescue you.

Our own version of his number, taken from the pairs where the first study came out significant: the replication's p value falls below 0.0004 five percent of the time and above 0.6791 five percent of the time. Same effect. Same design. Same code.

Rob said the tables were not working. He wanted twenty-four individual studies, printed out, one at a time. He was right. Three separate people who did not believe the 50% number have since been shown this figure.

A single p value is one draw from a wide distribution. Reporting it to three decimal places claims a precision the quantity does not have. We round to two now. And we say what the interval was.

Why the Wrong Versions Are So Appealing

None of these misreadings are stupid. Each is a reasonable answer to a reasonable question, attached to the wrong number. Understanding why they feel right is most of what stops you making them again.

The first one answers the question you asked. You did not run the study to learn about hypothetical data in a world you disbelieve; you ran it to find out whether the thing is real. \(P(\text{hypothesis} \mid \text{data})\) is the quantity your brain ordered. So a number turns up looking like a probability, sitting right beside a conclusion, and your brain signs for the delivery without reading the label. Goodman calls the resulting muddle the p-value fallacy, and traces it to a historical mashing-together of two incompatible systems: Fisher's evidential p and the Neyman-Pearson error-rate framework. Two systems never meant to appear in the same sentence [7, 17].

Small numbers feel like big claims, and that keeps the second one alive. Scientific writing has trained everybody to read \(p < 0.001\) as emphasis. Three zeros look like shouting. A grain of truth sits underneath, because at a fixed sample size a smaller p really does mean a bigger observed effect, and the grain of truth is exactly what makes the habit dangerous. Sample sizes are never fixed across the papers you are comparing.

Thresholds survive because decisions genuinely work that way. Somebody has to decide whether to fund the trial, a decision rule needs a line, and Neyman and Pearson designed their framework for exactly that, a procedure with known long-run error rates, inside which a threshold is the right tool [17]. The error comes from carrying the decision rule back into the evidence. The line tells you what to do next; what is true stays exactly as unsettled as it was before anybody drew it, and 0.051 remains very nearly identical evidence to 0.049.

Our language has no word for "we looked and could not tell". The fourth misreading fills the gap. Every report needs a finding. "No significant difference" sounds like a finding, reads like a finding, and gets summarised as a finding, where the accurate version says "our study was too small to distinguish a 30% benefit from a 10% harm". Terrible headline. Almost nobody writes it.

Every one of these is the mind doing something sensible with a number that was not built for it.

A fifth misreading deserves a mention, quieter and worse. People treat the p value as a property of the effect rather than of the study. The effect does not have a p value; the study has one, and a different study of the same effect would hand you a different one, as Figure 5 shows twenty-four times over.

The Strongest Case Against Everything Above

We should take the other side seriously, because the other side is not silly and includes people who have thought about this far longer than we have.

The argument runs like this. Yes, p values are misinterpreted. So is every statistic ever invented, including the ones proposed as replacements. Bayes factors depend on a prior over effect sizes, usually chosen by convention and rarely defended, and a reader who cannot interpret a p value will do no better with a quantity carrying an extra arbitrary input. Confidence intervals get misread as containing the true value with 95% probability, which is the same transposed conditional in a different coat, so the misinterpretation plainly lives in the reader. Move the reader to a new tool and the misinterpretation moves too.

More pointedly: the p value does one job honestly, and it is a job nothing else does as cheaply. It calibrates a test against a reference distribution and gives a number whose long-run behaviour under the null is known exactly, without requiring you to specify what you believe beforehand. That property is what makes regulatory decisions possible between parties who disagree about priors. A drug regulator and a drug company do not share a prior. They can share an alpha.

Now our own weakest joint. The base-rate argument in §8 depends on a number, \(R\), that nobody can measure. We chose 10% because it makes a good demonstration. A defender can reasonably answer that a well-motivated field, working from mechanism and earlier evidence, might run at \(R = 0.5\) or higher. Look that row up in the table. Only 9.1% of significant findings are wrong, which is roughly what a reasonable person expects anyway. Our headline number is a choice about \(R\) dressed up as a result. We should say so plainly. We just did.

The last objection has teeth. Reproducibility numbers are not clean evidence for the story we are telling. The Open Science Collaboration replicated 100 psychology studies and got 36% significant. Widespread false positives would produce that [18]. So would real but smaller effects. So would replications differing from the originals in ways that mattered. So would the plain power arithmetic in §9, under which honest replications of real effects fail routinely.

Where we come out. The strongest version of the defence is correct about the tool and wrong about the practice. Nothing in this article says the p value is a bad statistic. It says the p value is a narrow one, asked to carry a decision, an effect size, a measure of evidence and a claim about replication. Nobody built it for any of that. The ASA's 2016 statement lands in the same place, and it had to be negotiated among statisticians who agreed about almost nothing else. P values are useful. They do not measure the probability that a hypothesis is true, nor the size of an effect, nor the importance of a result, nor on its own how much evidence a model deserves [5]. We will keep using them. We will keep reading the confidence interval first.

What We Do Now, Concretely

Rules the club adopted for itself. We apply them to other people's papers too, which has made us harder to sit next to.

Report the estimate and its interval before the p value, in the same sentence. "The coffee group finished 1.7 minutes faster (95% CI 0.2 to 3.2, \(p = 0.031\))." Now the reader can see it. The data sit comfortably with an effect anywhere from trivial to substantial. Say so.

Never write "no effect" for a non-significant result. Write what the study could have detected. If the interval includes effects you would have cared about, the study did not settle the question.

If you do not know the study’s power, say so. Power depends on the effect size you were hoping to find, so state that too, because a study designed to detect \(d = 0.8\) has almost nothing to say about \(d = 0.2\).

The number of tests you ran belongs in the paper. Twenty independent tests under a true null give you a 64% chance of at least one result below 0.05, and the error rate that matters runs across every test you performed. Not the one you chose to write up [6].

Treat the threshold as a filing decision, not a conclusion. Amrhein, Greenland and McShane, with 800 co-signatories, argued for retiring the phrase "statistically significant" entirely [19]. We have not gone that far. We have stopped writing the word "significant" without a number next to it.

One last thing, and keep this one if you keep nothing else: the p value answers a question about data in a world where the effect is absent. Any sentence beginning "therefore the hypothesis is probably" has left that world without telling you. Go back and check what you actually computed.

References