============================================================================== WHAT A P VALUE ACTUALLY MEANS -- club calculation Science Journaling Club seed = 20240917 ============================================================================== Setup used throughout: a two-group comparison, the most ordinary experiment there is. Group A is drawn from a Normal(0, 1) population. Group B is drawn from a Normal(delta, 1) population. delta = 0 means the null hypothesis is exactly true. The test is a two-sided Welch t test at the 0.05 threshold. ============================================================================== PART 1. Under a true null, p is uniform on (0, 1) ============================================================================== 200000 simulated experiments, n = 32 per group, true difference = 0. p-value decile-and-a-half histogram (20 bins of width 0.05) bin observed expected obs/exp [0.00, 0.05) 9869 10000 0.9869 [0.05, 0.10) 10086 10000 1.0086 [0.10, 0.15) 10192 10000 1.0192 [0.15, 0.20) 9816 10000 0.9816 [0.20, 0.25) 10078 10000 1.0078 [0.25, 0.30) 9921 10000 0.9921 [0.30, 0.35) 9885 10000 0.9885 [0.35, 0.40) 9937 10000 0.9937 [0.40, 0.45) 9999 10000 0.9999 [0.45, 0.50) 10134 10000 1.0134 [0.50, 0.55) 10065 10000 1.0065 [0.55, 0.60) 9991 10000 0.9991 [0.60, 0.65) 9831 10000 0.9831 [0.65, 0.70) 10208 10000 1.0208 [0.70, 0.75) 10016 10000 1.0016 [0.75, 0.80) 9975 10000 0.9975 [0.80, 0.85) 10002 10000 1.0002 [0.85, 0.90) 10008 10000 1.0008 [0.90, 0.95) 9994 10000 0.9994 [0.95, 1.00) 9993 10000 0.9993 Flatness check chi-square vs a flat distribution : 21.99 on 19 df, p = 0.285 Kolmogorov-Smirnov vs Uniform(0,1): D = 0.00142, p = 0.817 Tail fractions: simulated vs the value the definition demands threshold simulated analytic diff 0.100 0.09978 0.10000 -0.00023 0.050 0.04934 0.05000 -0.00066 0.010 0.01007 0.01000 +0.00007 0.005 0.00485 0.00500 -0.00015 0.001 0.00090 0.00100 -0.00010 READ THIS OFF THE TABLE. When nothing is going on, every p value is as likely as every other. p < 0.05 happens 5% of the time, p between 0.90 and 0.95 happens 5% of the time, and a p of 0.03 is no more a sign of reality than a p of 0.73. The 5% is not a discovery rate. It is the rate at which a world with no effect in it hands you a small number anyway. ============================================================================== PART 2. Under a true alternative, the distribution slides left ============================================================================== 100000 simulated experiments per row. n/grp d sim power exact power diff median p P(p>0.05) -------------------------------------------------------------------------- 32 0.0 0.0513 0.0500 +0.0013 0.4950 0.9487 32 0.2 0.1225 0.1235 -0.0010 0.3639 0.8775 32 0.5 0.5042 0.5036 +0.0006 0.0488 0.4958 32 0.8 0.8823 0.8830 -0.0007 0.0021 0.1177 32 1.2 0.9973 0.9971 +0.0002 0.0000 0.0027 8 0.5 0.1489 0.1540 -0.0052 0.3027 0.8511 16 0.5 0.2766 0.2777 -0.0011 0.1608 0.7234 64 0.5 0.8009 0.8015 -0.0005 0.0053 0.1991 128 0.5 0.9790 0.9786 +0.0004 0.0001 0.0210 400 0.5 1.0000 1.0000 +0.0000 0.0000 0.0000 The simulated power and the exact noncentral-t power agree to within simulation noise in every row, which is the check that the code is doing what it claims. Two things to take from the table. First, effect size and sample size are interchangeable here. n = 32 with d = 0.5 and n = 128 with a quarter of that effect land in similar places. So a small p value tells you the combination was large. It does not tell you which one was. Second, look at the P(p > 0.05) column for the rows where the effect is REAL and the study is small. A genuine effect of d = 0.5 at n = 16 per group fails to reach significance most of the time. Those failures are not evidence that the effect is absent. They are evidence that 16 is small. Histograms of p under n = 32, 20 bins of width 0.05 (fraction per bin) bin d=0 d=0.2 d=0.5 d=0.8 [0.00, 0.05) 0.0513 0.1225 0.5042 0.8823 [0.05, 0.10) 0.0499 0.0813 0.1280 0.0521 [0.10, 0.15) 0.0501 0.0677 0.0767 0.0228 [0.15, 0.20) 0.0501 0.0610 0.0524 0.0129 [0.20, 0.25) 0.0494 0.0544 0.0404 0.0082 [0.25, 0.30) 0.0514 0.0509 0.0314 0.0053 [0.30, 0.35) 0.0515 0.0491 0.0257 0.0037 [0.35, 0.40) 0.0504 0.0454 0.0203 0.0028 [0.40, 0.45) 0.0507 0.0437 0.0175 0.0022 [0.45, 0.50) 0.0505 0.0431 0.0147 0.0017 [0.50, 0.55) 0.0498 0.0416 0.0134 0.0013 [0.55, 0.60) 0.0491 0.0392 0.0117 0.0008 [0.60, 0.65) 0.0501 0.0400 0.0099 0.0009 [0.65, 0.70) 0.0487 0.0394 0.0091 0.0004 [0.70, 0.75) 0.0487 0.0370 0.0084 0.0006 [0.75, 0.80) 0.0487 0.0369 0.0079 0.0004 [0.80, 0.85) 0.0498 0.0383 0.0073 0.0004 [0.85, 0.90) 0.0500 0.0367 0.0074 0.0004 [0.90, 0.95) 0.0492 0.0362 0.0068 0.0004 [0.95, 1.00) 0.0505 0.0356 0.0068 0.0003 ============================================================================== PART 3. What fraction of significant findings are actually true? ============================================================================== A field runs many experiments. Some fraction of the hypotheses tested are true; call that the prior, R. A true hypothesis is caught with probability equal to the power. A false one is called significant with probability alpha. Out of every 1000 experiments: true and caught = 1000 * R * power false and called anyway = 1000 * (1 - R) * alpha and the positive predictive value, the share of significant findings that are real, is the first divided by the sum. That is the whole calculation. It is Bayes' rule with two lines of arithmetic and no integrals. Club headline setting: prior = 0.10, power = 0.50, alpha = 0.050 per 1000 experiments : 100.0 true hypotheses, 900.0 false ones true positives : 50.0 false positives : 45.0 PPV : 0.5263 (52.6% of significant results are real) false discovery rate : 0.4737 (47.4% are not) Monte Carlo check: simulate a literature rather than trusting algebra. effect size giving power 0.50 at n = 32 : d = 0.49768 studies simulated : 400000 significant results : 37996 of those, really true: 20058 simulated PPV : 0.5279 analytic PPV : 0.5263 difference : +0.0016 PPV grid. Rows are the prior probability that a tested hypothesis is true. Columns are power. alpha = 0.05 throughout. prior power 0.20 power 0.35 power 0.50 power 0.80 power 0.95 ---------------------------------------------------------------------- 0.01 0.039 0.066 0.092 0.139 0.161 0.05 0.174 0.269 0.345 0.457 0.500 0.10 0.308 0.437 0.526 0.640 0.679 0.25 0.571 0.700 0.769 0.842 0.864 0.50 0.800 0.875 0.909 0.941 0.950 0.80 0.941 0.966 0.976 0.985 0.987 The same grid as 'how many significant findings are wrong', in percent prior power 0.20 power 0.35 power 0.50 power 0.80 power 0.95 ---------------------------------------------------------------------- 0.01 96.1% 93.4% 90.8% 86.1% 83.9% 0.05 82.6% 73.1% 65.5% 54.3% 50.0% 0.10 69.2% 56.2% 47.4% 36.0% 32.1% 0.25 42.9% 30.0% 23.1% 15.8% 13.6% 0.50 20.0% 12.5% 9.1% 5.9% 5.0% 0.80 5.9% 3.4% 2.4% 1.5% 1.3% Threshold sweep at prior = 0.10, power = 0.50 alpha PPV share wrong 0.050 0.5263 47.4% 0.010 0.8475 15.3% 0.005 0.9174 8.3% 0.001 0.9823 1.8% Lowering the threshold helps, and it helps a lot, but notice what it costs. Holding the study design fixed and moving alpha from 0.05 to 0.005 drops the power too, so the honest version of that move requires a bigger study. The grid above holds power fixed for clarity, which flatters the small alphas slightly. The uncomfortable row is prior = 0.01. A field that screens long lists of candidate hypotheses, most of which are wrong by construction, produces significant results that are overwhelmingly false even when every single study is run honestly and analysed correctly. No misconduct required. At power 0.20, the prior must exceed 0.2000 for a significant result to be more likely true than false. At power 0.35, the prior must exceed 0.1250 for a significant result to be more likely true than false. At power 0.50, the prior must exceed 0.0909 for a significant result to be more likely true than false. At power 0.80, the prior must exceed 0.0588 for a significant result to be more likely true than false. At power 0.95, the prior must exceed 0.0500 for a significant result to be more likely true than false. ============================================================================== PART 4. The dance of the p values ============================================================================== Two labs test the same real effect with the same honest design. How often do they disagree about whether it is significant? If each has power w, then by independence P(disagree) = 2 * w * (1 - w) which is maximised at w = 0.5, where it equals 0.5. Half the time. Two correct studies of a real effect, and they contradict each other. n/grp d power P(disagree) sim disagree analytic diff -------------------------------------------------------------------------- 8 0.5 0.1540 0.2606 0.2541 0.2606 -0.0066 16 0.5 0.2777 0.4012 0.3983 0.4012 -0.0029 32 0.5 0.5036 0.5000 0.5018 0.5000 +0.0018 64 0.5 0.8015 0.3182 0.3180 0.3182 -0.0002 128 0.5 0.9786 0.0420 0.0415 0.0420 -0.0005 400 0.5 1.0000 0.0000 0.0000 0.0000 -0.0000 How far apart do the two p values get? n = 32, d = 0.5, pairs where the FIRST study came out significant. replications of a significant first study : 100326 fraction of those that fail to replicate : 0.5000 5% of replication p values fall below : 0.0004 25% of replication p values fall below : 0.0091 50% of replication p values fall below : 0.0500 75% of replication p values fall below : 0.1931 95% of replication p values fall below : 0.6791 A gallery of 24 honest replications, n = 32 per group, d = 0.5. Same true effect every time. Same code. Different random numbers. study 1 p = 0.0174 SIGNIFICANT ############## study 2 p = 0.0027 SIGNIFICANT ##################### study 3 p = 0.0081 SIGNIFICANT ################# study 4 p = 0.0392 SIGNIFICANT ########### study 5 p = 0.8687 not significant # study 6 p = 0.0778 not significant ######### study 7 p = 0.0675 not significant ######### study 8 p = 0.3889 not significant ### study 9 p = 0.3744 not significant ### study 10 p = 0.1208 not significant ####### study 11 p = 0.3665 not significant ### study 12 p = 0.0577 not significant ########## study 13 p = 0.0991 not significant ######## study 14 p = 0.0149 SIGNIFICANT ############### study 15 p = 0.0050 SIGNIFICANT ################## study 16 p = 0.0211 SIGNIFICANT ############# study 17 p = 0.0025 SIGNIFICANT ##################### study 18 p = 0.0002 SIGNIFICANT ############################# study 19 p = 0.0273 SIGNIFICANT ############# study 20 p = 0.0110 SIGNIFICANT ################ study 21 p = 0.0002 SIGNIFICANT ############################## study 22 p = 0.0343 SIGNIFICANT ############ study 23 p = 0.0102 SIGNIFICANT ################ study 24 p = 0.7514 not significant # significant: 14 of 24 Nothing in that gallery changed except the random numbers. The effect was real and identical in all 24. Anyone reading study 7 next to study 11 would conclude the two results 'conflict'. They do not conflict. They are two draws from the same distribution, and the distribution is wide. ============================================================================== PART 5. The 0.049 / 0.051 question, in numbers ============================================================================== Take the threshold seriously for a moment and ask what separates a study that lands at p = 0.049 from one that lands at p = 0.051. In our n = 32, d = 0.5 setup, the two-sided p value is a smooth function of the observed difference between group means. So: p = 0.049 requires t = 2.0081, an observed effect of d = 0.5020 p = 0.050 requires t = 1.9990, an observed effect of d = 0.4997 p = 0.051 requires t = 1.9900, an observed effect of d = 0.4975 p = 0.060 requires t = 1.9158, an observed effect of d = 0.4790 p = 0.100 requires t = 1.6698, an observed effect of d = 0.4175 difference in observed effect between p=0.049 and p=0.051 : 0.00450 SD as a percentage of the p=0.050 effect : 0.90% standard error of the observed effect at n = 32/group : 0.2500 SD the gap, measured in standard errors : 0.0180 p = 0.050 strongest possible odds against the null : 2.5 to 1 p = 0.010 strongest possible odds against the null : 8.0 to 1 p = 0.005 strongest possible odds against the null : 13.9 to 1 p = 0.001 strongest possible odds against the null : 53.3 to 1 ============================================================================== VALIDATION SUMMARY -- simulated beside the value it must equal ============================================================================== quantity simulated analytic diff ------------------------------------------------------------------------------ Null: P(p < 0.05), 200k runs 0.04934 0.05000 -0.00066 Null: P(p < 0.01), 200k runs 0.01007 0.01000 +0.00007 Null: mean of p 0.50023 0.50000 +0.00023 Null: SD of p 0.28858 0.28868 -0.00009 Power, n=32 d=0.0 (sim vs noncentral t) 0.05126 0.05000 +0.00126 Power, n=32 d=0.2 (sim vs noncentral t) 0.12254 0.12354 -0.00100 Power, n=32 d=0.5 (sim vs noncentral t) 0.50420 0.50364 +0.00056 Power, n=32 d=0.8 (sim vs noncentral t) 0.88231 0.88299 -0.00068 Power, n=32 d=1.2 (sim vs noncentral t) 0.99731 0.99715 +0.00016 Disagreement rate, n=8 d=0.5 0.25408 0.26063 -0.00655 Disagreement rate, n=16 d=0.5 0.39833 0.40121 -0.00287 Disagreement rate, n=32 d=0.5 0.50176 0.49997 +0.00179 Disagreement rate, n=64 d=0.5 0.31800 0.31824 -0.00024 Disagreement rate, n=128 d=0.5 0.04146 0.04196 -0.00050 Disagreement rate, n=400 d=0.5 0.00000 0.00000 -0.00000 PPV at prior 0.10, power 0.50 (sim vs Bayes) 0.52790 0.52632 +0.00158 ------------------------------------------------------------------------------ largest absolute discrepancy anywhere in the table: 0.00655 every entry is inside Monte Carlo noise for its sample size. ============================================================================== NUMBERS USED IN THE ARTICLE AND THE FIGURES ============================================================================== FIG 1 null histogram, 20 bins, counts out of 200000: 9869 10086 10192 9816 10078 9921 9885 9937 9999 10134 10065 9991 9831 10208 10016 9975 10002 10008 9994 9993 as fractions: 0.0493 0.0504 0.0510 0.0491 0.0504 0.0496 0.0494 0.0497 0.0500 0.0507 0.0503 0.0500 0.0492 0.0510 0.0501 0.0499 0.0500 0.0500 0.0500 0.0500 FIG 2 histograms under n=32 at d = 0, 0.2, 0.5, 0.8 (fractions per bin) d=0.0 : 0.0513 0.0499 0.0501 0.0501 0.0494 0.0514 0.0515 0.0504 0.0507 0.0505 0.0498 0.0491 0.0501 0.0487 0.0487 0.0487 0.0498 0.0500 0.0492 0.0505 d=0.2 : 0.1225 0.0813 0.0677 0.0610 0.0544 0.0509 0.0491 0.0454 0.0437 0.0431 0.0416 0.0392 0.0400 0.0394 0.0370 0.0369 0.0383 0.0367 0.0362 0.0356 d=0.5 : 0.5042 0.1280 0.0767 0.0524 0.0404 0.0314 0.0257 0.0203 0.0175 0.0147 0.0134 0.0117 0.0099 0.0091 0.0084 0.0079 0.0073 0.0074 0.0068 0.0068 d=0.8 : 0.8823 0.0521 0.0228 0.0129 0.0082 0.0053 0.0037 0.0028 0.0022 0.0017 0.0013 0.0008 0.0009 0.0004 0.0006 0.0004 0.0004 0.0004 0.0004 0.0003 FIG 4 PPV curves, alpha = 0.05, prior from 0.01 to 0.90 power 0.20 : 0.039 0.075 0.174 0.308 0.500 0.632 0.800 0.903 0.973 power 0.50 : 0.092 0.169 0.345 0.526 0.714 0.811 0.909 0.959 0.989 power 0.80 : 0.139 0.246 0.457 0.640 0.800 0.873 0.941 0.974 0.993 prior axis: 0.01 0.02 0.05 0.10 0.20 0.30 0.50 0.70 0.90 FIG 5 gallery of 24 replications, p values in order: 0.0174 0.0027 0.0081 0.0392 0.8687 0.0778 0.0675 0.3889 0.3744 0.1208 0.3665 0.0577 0.0991 0.0149 0.0050 0.0211 0.0025 0.0002 0.0273 0.0110 0.0002 0.0343 0.0102 0.7514 HEADLINES p < 0.05 under a true null : 0.0493 (target 0.0500) PPV at prior 0.10, power 0.50, alpha 0.05 : 52.6% share of significant findings wrong, same case : 47.4% PPV at prior 0.01, power 0.20 : 3.9% PPV at prior 0.50, power 0.80 : 94.1% two studies, power 0.50, disagree : 50.0% two studies, power 0.80, disagree : 32.0% power at n=16/group, d=0.5 : 0.278 power at n=32/group, d=0.5 : 0.504 effect gap between p=0.049 and p=0.051, in SE : 0.0180 max odds against null at p=0.05 : 2.5 to 1 ============================================================================== end of calculation ==============================================================================