How Many Reasonable Choices Does It Take to Find a Result That Is Not There

Seventy-One Studies in a Hundred

We have done this. That is the reason for the article, and it seems fairer to say so at the top than to let it surface in a footnote. Two years ago a few of us ran a small survey for a club project, looked at the outcome we had planned to look at, found nothing, and then kept going. We tried the other question on the form. Then we dropped two people who had obviously not read the instructions. Then we looked at the younger half separately, because someone suggested the effect might be stronger there, and then at the older half for symmetry. Somewhere in that sequence a p value came out at 0.03 and we wrote it up. Nobody in the room believed we had cheated, because at every step there had been a reason, and the reason had been a good one.

This study is what happens when you run that sequence on data you know to be empty.

We simulated 50,000 experiments in which two groups were drawn from exactly the same distribution. No effect exists anywhere in them. Every significant result is false by construction, which is the whole point: it means we can count the errors exactly instead of arguing about them. Analysed once, as planned, at the sample size planned, 5.04% of those studies came out significant, which is what the 5% level promises. Analysed with five ordinary decisions left open until after the data arrived, 70.66% came out significant.

Seventy-one studies in a hundred. Not because anybody faked anything, and not because any individual test was wrong. Each of the 180 analyses available to the last row of that table is a defensible analysis of the data. The damage is done entirely by choosing among them after looking.

What follows sets out the model, spends a whole section on the one number that certifies the rest, measures all 32 combinations of the five freedoms, checks the answer against two published results from 1969 and 2011, and then names the places where our own choices moved the headline.

What Is Actually Being Simulated

The club has no laboratory, no participants and no ethics approval, and none of those things would have helped here. Everything below came out of one Python file. It has no knowledge of psychology, medicine or any other subject in which this failure mode does damage. It knows about normal distributions and t statistics. The computation is the experiment, and the word "study" below always means a pair of number lists, never a person.

One simulated study contains the following. Two groups of 40 at full enrolment, labelled treatment and control, where the label affects nothing whatsoever. Three outcome measures per participant, jointly normal with mean zero, variance one, and pairwise correlation 0.5, which stands in for three related scales on the same questionnaire. One covariate per participant, standard normal, independent of everything including the outcomes, standing in for a baseline measurement somebody suspects might matter. One binary attribute per participant, a fair coin, standing in for a demographic split with no bearing on anything.

Five researcher degrees of freedom act on that study:

A combination of freedoms counts as having found something if any analysis it permits returns a two-sided p below 0.05. Turning all five on permits \(5 \times 3 \times 3 \times 2 \times 2 = 180\) paths through the same data set. None of the 180 is a bad test. Every one of them would pass a reviewer on its own, described on its own, in a paper that did not mention the other 179.

What the model leaves out is larger than what it contains. There is no file drawer, no choice of statistical test, no transformation of the outcome, no dropping of an entire condition, no hypothesis invented after the fact [5]. Wicherts and colleagues enumerate 34 decision points in an ordinary psychology study [7]; we modelled five of them. There is also no genuine effect anywhere, so nothing here speaks to statistical power, to bias in the size of a real effect, or to what these same practices do when something true is present. The number 70.66% is an answer about five specific freedoms exercised exhaustively, and it is a floor rather than a ceiling with respect to the full list.

The Baseline Is the Instrument

Everything in this article is a rate measured off a simulation. A simulation with a bug in its t test would produce rates too, and they would look exactly as convincing. So before any result is reported the file has to demonstrate that when nothing is being done wrong, nothing comes out wrong.

Three checks, printed in the raw output in this order.

The test itself. Our two-sample t statistic and its p value are computed from weighted sums of powers, with the tail probability coming from an incomplete beta function written out in the file rather than imported. On 20,000 identical data sets it was placed beside scipy.stats.ttest_ind. Largest disagreement in t: 1.332×10⁻¹⁵. Largest disagreement in p: 1.193×10⁻¹¹. Studies on which the two routines disagreed about significance: zero. The covariate-adjusted statistic, which solves the normal equations directly, was placed beside numpy.linalg.lstsq, a different LAPACK driver sharing no code with it, with the tail probability from scipy. Largest disagreement in t over 500 studies: 3.553×10⁻¹⁵.

A case small enough to check with a pencil. Group A is 4, 7, 5, 6, 8. Group B is 6, 9, 7, 10, 8. Both variances are 2.5, so the pooled variance is 2.5, the standard error is \(\sqrt{2.5 \times 0.4} = 1\), and t is exactly \(-2\) on 8 degrees of freedom. Our p is 0.080516238. Scipy's is 0.080516238. Anybody who suspects the machinery can redo that line by hand in about four minutes.

The baseline rate. One outcome, one test, at the planned sample size, nothing chosen after the fact. Out of 50,000 studies, 2,522 came out significant.

Under 0.05 in the long run it must be. It is, to within half a standard error, and the interval covers the nominal value comfortably. We also checked the shape rather than only the tail: under a true null the p values must be uniform on the unit interval, and the ten deciles of our 50,000 baseline p values gave a chi-square of 6.45 on 9 degrees of freedom against a 5% critical value of 16.92. A test that was subtly wrong in the middle of its distribution would show up there even if the 5% tail happened to come out right.

If that baseline had come back at 0.06, nothing further in this article would be worth reading, and we would have spent the term looking for the mistake instead of writing it up.

Working Notes From the Club Table

The Arithmetic

Three calculations carry the whole paper, and none of them needs a computer.

First, the standard error on a rate. Each study either comes out significant or does not, so a rate over \(n\) studies is a binomial proportion and its standard error is \(\sqrt{\hat p (1-\hat p)/n}\). For the headline, \(\hat p = 0.70662\) and \(n = 50{,}000\):

which is why the article quotes 0.7066 and not 0.70662. Four decimal places is already more than the run supports.

Second, the effective number of independent tests. If somebody ran \(k\) tests that were genuinely independent of each other, each at the 5% level, the chance that at least one came out significant would be \(1 - 0.95^{k}\). Run that backwards. Given an observed rate, the \(k\) that would produce it is

and for the headline that is:

So 180 paths purchase the damage of about 24 independent coin flips. Compare that with what 180 independent tests would actually do: \(1 - 0.95^{180} = 1 - 9.778\times10^{-5} = 0.999902\). Had the paths been independent the false positive rate would have been 99.99% rather than 70.7%. They are not independent, because they are 180 views of the same 80 numbers, and that overlap is the only thing standing between the practice and total failure.

Third, the accounting. Take the five freedoms in the order S, M, G, C, O and ask, for each study, at which point it first became significant. Every study falls into exactly one bucket, and the buckets have to add up.

Read the bucket marked 2. Fifteen and a half percent of all studies of pure noise were rescued by the second thing a person tries, which is looking at the other outcome measure they happened to collect. That bucket alone is three times the honest error rate.

Five Freedoms, Taken One at a Time

Figure 1 puts all 32 combinations in one frame, arranged by how many freedoms are in play.

Singly, the order of damage is outcome choice at 0.1286, then subgroup analysis at 0.1165, then optional stopping at 0.1139, then outlier exclusion at 0.0736, and last the covariate at 0.0548. The covariate is nearly free. Adding or dropping an uninformative covariate changes the p value hardly at all, because the covariate explains no variance and the t statistic barely moves, so the second test is almost the same test. This was the one result that surprised the room, since covariate fishing has a bad reputation it does not, on these numbers, deserve on its own. In combination it is a different matter, because a nearly-free extra path is still an extra path and it multiplies against everything else.

The two heavy lines in Figure 1 are the same five freedoms adopted in opposite orders. They start together at 0.0504 and finish together at 0.7066, and in between they are nowhere near each other. Adopt stopping first and you are at 0.114 after one step; adopt outlier exclusion first and you are at 0.074. Three steps in, the two orders sit at 0.521 and 0.185. Whichever freedom you take first appears to be the expensive one, and that is an artefact of ordering rather than a fact about the freedom.

Three freedoms is where it stops being an inflation and starts being a coin toss. S, M and G together give 0.5206, so a study of pure noise is more likely than not to produce a publishable p value. That combination needs no unusual behaviour at all. It needs somebody who glances at the numbers while they are still coming in. It needs a questionnaire carrying more than one scale. And it needs the ordinary, sensible-sounding question of whether the result holds up in each half of the sample.

Nineteen Sixty-Nine, and Twenty Eleven

None of this is new. Armitage, McPherson and Rowe worked out the cost of repeated significance testing on accumulating data in 1969 and published a table of it [2]. Simmons, Nelson and Simonsohn ran the modern simulation in 2011 and published a table of that [1]. If our code is right it has to reproduce both.

The 1969 result is the cleanest external check available, because it is a table of numbers with no room to argue. It assumes equally spaced looks at accumulating data, so we built that setting separately rather than reusing the main grid, with 25 new participants per group between looks, 200,000 simulated studies, and critical values from our own t quantile function.

Every published entry is reproduced. The largest deviation anywhere in the table is 2.67 standard errors, at twenty looks, where their published value is rounded to three decimals and ours is 0.24858 against 0.246. We report that as agreement rather than as a discrepancy, because the rounding alone is worth 0.0005 and our standard error there is 0.00097, so the published figure is consistent with anything between 0.2455 and 0.2465. At five looks the two agree exactly to the three decimals the 1969 table prints, which is luck and nothing more.

The 2011 paper is harder to check and more worth checking, because it is the paper this study grew out of. Its Table 1 gives false positive rates for four researcher degrees of freedom at three significance levels, from 15,000 simulations each. Three of the four can be built out of pieces already in our file. The fourth needs a three-condition design, which our model does not have, so we did not attempt it. Every operationalisation was taken from the note printed under their table rather than from our reading of the text, which matters: Situation A turns out to be three t tests, one on each of two correlated outcomes and a third on their average, and we would have guessed two.

Fifteen comparisons, largest deviation 2.67 standard errors, and the sigma column already includes their Monte Carlo error as well as ours, since 15,000 simulations carry a standard error of about 0.0038 on a rate near 0.3. The consistent small positive sign across the table is worth noticing and we cannot explain it; it may be the interaction coding described below, or it may be that their simulations and ours differ in some detail the published note does not record.

Both checks matter for a different reason than they appear to. Neither is a discovery. What they establish is that a club with a laptop and no supervision can reproduce two independent published results to within their own Monte Carlo error, which is the only reason anybody should take the 70.66% seriously, since that number has nothing published to check it against.

Watching It Settle

A rate measured off 50,000 draws is a measurement with an error bar, and error bars from a Monte Carlo run have a habit of being quoted without anybody checking that the run was long enough to earn them. So here is the whole history.

At 100 studies the baseline estimate is 0.030 and the interval runs from below zero to 0.064, which is to say the run knows nothing. At 1,015 studies it is 0.0542. At 10,299 it is 0.0496. From about 20,000 onward it moves in the fourth decimal place and the interval narrows around 0.05 without moving off it. The full-freedom trace is flatter still, because a rate near 0.7 has a smaller relative error than a rate near 0.05 for the same number of draws. The first 10,000 studies give 0.70830 and the last 10,000 give 0.70990, which differ by less than one standard error of either half.

A trace still wandering at the right-hand edge would mean we had stopped too early. Neither of these is.

One Hundred and Eighty Paths Behave Like Twenty-Four

The gap between 180 paths and 23.9 effective tests is the most interesting number in the study, and it cuts both ways.

Every point sits below the diagonal, and the gap widens as paths multiply. Three outcome measures give 2.68 effective tests, which is 89% efficient, because three outcomes correlated at 0.5 really are close to three separate looks. Five sequential peeks give 2.36, which is 47% efficient, because a test at n = 25 and a test at n = 30 share 25 of their 30 observations and mostly agree with each other. By the time all five freedoms are on, 180 paths deliver 23.9 effective tests, an efficiency of 13%, or 7.5 paths for every test's worth of damage.

The optimistic reading is that redundancy is protective. If you could search 180 independent hypotheses you would be certain to find something, and nobody can, because the data will not support that many genuinely different questions. The pessimistic reading is that 24 independent tests at the 5% level is a catastrophe by any standard, and that the protection saturates: going from 90 paths to 180 bought the searcher only another 7.7 effective tests, but it still moved the false positive rate from 0.565 to 0.707.

There is a practical consequence. A Bonferroni correction applied to the number of analyses a researcher actually reports is the wrong correction, and it is wrong in both directions at once. It is too strict if the analyses overlap heavily, as ours do. It is far too lenient if the analyses reported are a selection from a larger set that was run and not reported, which is the situation this article is about. The quantity you would need is \(k_{\text{eff}}\) over the paths actually walked, and nobody has that number for a real paper, including the authors of the paper.

The Strongest Objection We Could Make

The objection runs like this.

Nothing in this article is a discovery. That selecting the smallest of several p values inflates the false positive rate has been known since before any of us was born, has a closed-form answer in the independent case, and was measured in exactly this way, on exactly this kind of simulation, in a paper published in 2011 that we cite in our own abstract [1]. The club has reproduced a fifteen-year-old result with more replicates and a green stylesheet. Worse: the researcher in our model is a machine that exhaustively enumerates 180 analyses and returns the minimum, which no human being does. Real researchers stop when they find something, or when the afternoon runs out. The 70.66% describes a search nobody performs.

We think most of that is correct. Here is what survives it.

The exhaustive searcher is the right upper bound for the freedoms we modelled and the wrong bound for the practice as a whole, and those two errors point in opposite directions. A real researcher walks fewer of our 180 paths, which pushes the real rate down. A real researcher also has access to freedoms we did not model at all, of which Wicherts and colleagues list dozens [7], which pushes it up. We cannot say which dominates, and we say so instead of implying that 70.66% is the rate for a real field. It is the rate for this model, exercised exhaustively, and the interactive version of this article exists partly so that a reader can set the number of paths themselves and watch what a less thorough searcher achieves.

The reproduction of 1969 and 2011 is not a discovery and was never meant to be one. It is the thing that licenses the rest. Anybody can print a number; a number that agrees with somebody else's number, computed by different people from a different program in a different decade, is a different kind of object. Our 15 comparisons against the 2011 table cost about thirty seconds of computer time and are the reason we believe our own machinery.

What we would defend hardest is the \(k_{\text{eff}}\) column. We have not seen the ratio of effective to nominal paths measured across a factorial grid of freedoms, and it is the part of the study that changed our own understanding. Before running it, several of us assumed that more analysis paths meant proportionally more risk. The measurement says that risk grows roughly with the logarithm of path count once overlap sets in, and that the overlap is doing more work to protect the literature than any correction procedure is. That is a modest thing to add, and it is measured rather than asserted, and it belongs to us.

A second objection we cannot dispose of. All 32 combinations were evaluated on one shared pool of 50,000 studies rather than on 32 independent pools. Each rate is still an unbiased estimate with a valid binomial standard error, so every number in the results table stands on its own. The rows are correlated with each other, though, so the differences between rows are more precise than independent runs would be, and the 32 rows are not 32 independent replications of anything. The sensitivity runs in the next section are independent, and the one that repeats the main settings came back at 0.70630 against the main pool's 0.70662, a difference of 0.1 standard errors, which is the only direct evidence we have on this point.

Where a Different Choice Would Have Changed the Answer

We made about a dozen modelling decisions. Four of them move the headline enough to matter, and a reader who takes 70.66% away from this article without taking the following four caveats has taken the wrong thing.

The correlation between outcome measures. We set it at 0.5 because that is what the 2011 paper used, and it is a plausible figure for two scales on one questionnaire. It is also completely unmeasured. At r = 0 the three outcomes are three independent tests and outcome choice alone costs 0.1420, with the full combination reaching 0.7525. At r = 0.8 the three outcomes are nearly one outcome, the same freedom costs 0.0986, and the full combination falls to 0.6132. A researcher whose outcome measures agree with each other is genuinely safer, and a researcher who collects deliberately varied measures is genuinely more exposed. The headline moves by fourteen percentage points across a range of correlations that nobody reports.

The outlier threshold. Two standard deviations is a common rule and it is the most generous of the common rules. It drops 4.55% of a normal sample, so roughly four of eighty observations, and it is worth 0.0736 on its own. Two and a half drops 1.24% and is worth 0.0654. Three drops 0.27% and is worth 0.0502, which is to say nothing at all: at three standard deviations the outlier rule almost never changes which side of 0.05 you land on. The full combination falls from 0.703 to 0.573 as the rule tightens from two to three. If we had written the file with a 3 SD default, the headline of this article would have been 57% and every sentence around it would have been the same.

The peek schedule. Our five looks run from 20 to 40 per group, adding 20 participants in total, and cost 0.1139. Armitage's five looks each add a full cohort, running from 25 to 125, and cost 0.1420. The ratio is 1.25. Both are accurately described by the sentence "we peeked five times". They are not the same act and they do not carry the same price, and any account of optional stopping that quotes an inflation figure without stating the schedule is quoting a number that could be off by a quarter or more. The published sequential-analysis literature has always known this, which is why Pocock's boundaries and the O'Brien and Fleming boundaries are specified in terms of information accrued rather than in terms of how many times somebody looked [13][14].

Fixing total enrolment. Holding the final sample at 40 per group in every arm of the study was chosen so that the optional-stopping effect could not be confused with a sample size effect. It also means our stopping rule never lets a researcher rescue a study by collecting more data, which is the commonest real version of the practice. A design in which peeking extends enrolment would give a larger stopping cost and a larger headline.

What we would tell ourselves two years ago

Not "be more careful", which is useless advice, and not "you cheated", which is false. The useful version is narrower. Write down the analysis before the data arrive, because the only difference between the honest 5.04% and the dishonest 70.66% is whether the decisions were made before or after looking [17]. Where that is impossible, count the paths out loud: say in the paper how many outcomes were collected, how many subgroups were examined, whether enrolment stopped at a planned point, and what the exclusion rule was before it was applied. A reader who knows you had 180 paths available can discount accordingly. A reader who is shown one path cannot.

And when a result appears only after the fourth thing you tried, treat the sequence itself as data about the result. In our simulation, 14.14% of all studies became significant only at the fifth freedom. Every one of those is false. The researcher who found them had, at that moment, a p value below 0.05 and a defensible justification for every step that got them there.

The broader literature on what this does to a field is large and more pessimistic than we are [3][4][6][16][18]. Simulation work has shown that a field whose incentives reward significant findings will drift towards exactly these practices with nobody intending it [15], and methods now exist for detecting the footprint they leave in a published literature after the fact [12]. The reproducibility programme of the last decade [11] and the many-analyst experiments [9][10] are downstream of the same observation, as are the multiverse proposals that ask an author to report every path rather than one [8]. We have nothing to add to any of it beyond our own numbers and our own admission.

Reproducing this

Everything in this article comes from one file and one command. You need Python 3.12 and numpy. Scipy is used only in the validation sections, and the file says so plainly if it is missing.

python forking-paths.py > forking-paths-output.txt

Expect a little over six minutes on a modern laptop. Ours took 377.9 s on numpy 2.4.2 and Python 3.12.3, of which the main pool of 50,000 studies was 62.8 s, the Armitage comparison at 200,000 studies was 5.6 s, and the 2011 replication at 120,000 studies per row was 37.6 s. The master seed is 20260214, hard coded at the top of the file, and every stream is spawned from it through numpy's SeedSequence, so the whole output is deterministic. Numbers differing from ours by more than the printed standard errors mean something is wrong, and we would like to hear about it. The interactive model runs a smaller version of the same experiment in your browser and lets you make the choices yourself, one at a time, on a single study.

References