The Treatment That Works Because You Picked the Worst Cases

Nothing Happened and They Got Better

Here is the entire experiment. Six million simulated people. Each one has a fixed underlying quality, a number that never changes. We measure each of them once, imperfectly. We take the six hundred thousand who scored worst. We do nothing to them. No pill, no training, no attention, no encouragement, no time passing, no illness running its course. In the model, literally nothing happens between the first measurement and the second, and we know this for certain because we wrote the program.

Then we measure them again. They are better by 0.8760 standard deviations.

That is a very large effect. Psychologists and clinicians call anything above 0.8 a large effect size, and it is the sort of number that gets a programme funded. Measured against the spread of the selected group itself, which is the comparison a before-and-after paper usually reports, the gain is 2.13 standard deviations. Nothing produced it. The selection produced it.

The name for this is regression to the mean, and everybody who has taken a statistics course has heard of it. What we wanted was the size. Not the concept, the number: how much of an ordinary-looking improvement can this supply on its own, and under what conditions does it supply all of it?

The answer turns out to depend on two things and nothing else. How extreme a tail you select, and how much of your measurement is noise. Both of those are decisions someone makes, usually early, usually without thinking of them as decisions about the size of their eventual result.

The Model, and What It Leaves Out

The club has no laboratory and no patients. We have a room with computers in it, so our research is computation. Everything below came out of a program we wrote, which runs a mathematical model and knows nothing about people. Nothing here was observed in anybody. When we say measured, we mean measured from our own simulated runs, the way you would measure the frequency of heads by flipping a coin many times rather than by reasoning about coins.

The model is as small as we could make it. Each individual carries a trait \(T\), drawn once from a normal distribution and then fixed forever. Each time we measure them, fresh independent noise is added:

The noise-to-signal ratio is \(\lambda = \sigma_e / \sigma_T\). The correlation between the two measurements, which in the measurement literature is called test-retest reliability, follows from it:

We fix the variance of a single measurement at 1, so every number in this article is already in units of the population standard deviation on one testing occasion. That is the unit a practitioner sees. Selection takes the bottom fraction \(p\) on \(X_1\). Nothing is applied. We report \(\Delta = \text{mean}(X_2 - X_1)\) over the selected group.

Now the part that matters more than the model. The trait is perfectly stable. There is no natural history, no recovery, no ageing, no learning, no seasonal cycle, no placebo response, no attention effect, no dropout, no floor or ceiling on the scale. Noise is normal, additive, the same size on both occasions, and independent between them. Real second measurements are almost never as pure as this. Anything that genuinely changes between the two occasions adds to \(\Delta\), and our model has no way of separating it out, because in our model there is nothing to separate.

Normality is doing more work than it looks. The shrinkage law we validate against is a property of the bivariate normal, and the truncated-mean factor that sets the size of the effect is a property of the normal tail. A heavy-tailed measurement changes both. We say where in §11.

The Arithmetic

This section contains no simulation. It is the whole result worked by hand, and a reader with a normal table can check every line of it.

Two measurements of the same people, each the same trait plus independent noise, are jointly normal with correlation \(r\). Both have mean 0 and variance 1. For a bivariate normal, the conditional expectation is linear, and with standardised marginals its slope is exactly \(r\):

Read that slowly. It says the expected second score of anybody is their first score multiplied by \(r\), a number less than one. So whatever group you pick using the first measurement, and by whatever rule, the group's mean on the second measurement is its mean on the first pulled toward the population mean by the factor \(r\). Averaging over any selected set \(S\) chosen from \(X_1\) alone,

Now choose the rule. Take the bottom fraction \(p\), which cuts at the quantile \(z_p = \Phi^{-1}(p)\). The mean of a standard normal truncated below \(z_p\) is a textbook quantity, \(-\varphi(z_p)/p\). Substituting and subtracting:

Two factors, multiplied. The first, \(1 - r\), is the share of the measurement that is noise. The second, \(\varphi(z_p)/p\), is how far out the tail you went. Work the headline cell:

The two agree to 1.2 standard errors. There is a second way to say the same thing that we found more useful at the club table than the formula. The factor \(1 - r\) is the fraction of the group's distance from the population mean that closes by itself. With \(r = 0.5\), the selected group is 1.755 SDs below average and half of that gap, 0.8775 SDs, evaporates. Not because anything moved. Because half of what put them in the group was noise, and noise does not come back.

Everything else in this article is that one product, evaluated at different places, checked against simulation, and then pointed at study designs.

Working Notes From the Club Table

What the Cut Does to a Population

Figure 1 is one cell drawn out. Six hundred individuals from the headline run, first measurement against second, with the bottom tenth shaded.

The figure is worth staring at, because it shows why the effect is not a trick of arithmetic but a consequence of cutting. The cloud of points is symmetric about the diagonal. Nobody is systematically improving. But the vertical strip we selected was chosen by its \(X_1\) coordinate, and within that strip the \(X_2\) values are drawn from the whole conditional distribution, which is centred at \(0.5 X_1\) rather than at \(X_1\). The dots in the strip sit, on average, above the dashed line. They were always going to.

Put the other way round: to be in the bottom tenth on a noisy measurement you need either a genuinely low trait or an unlucky day, and most people who qualify had some of both. The trait stays. The unlucky day does not.

The Grid

Six selection thresholds by seven noise ratios. Forty-two cells, six million simulated individuals each, 252 million in the grid. Each cell is 60 independent replicates of 100,000 people, and the standard error is the spread across those 60 runs rather than a formula we assumed.

Read the figure as a menu of design choices. A study that enrols the worst quarter on a measurement with reliability 0.8 buys itself 0.254 SD before anything happens. Move to the worst 1% on the same instrument and it is 0.530. Keep the worst quarter but use an instrument with reliability 0.31 and it is 0.880. None of those three studies has done anything wrong. None of them has done anything at all.

The bolded row is the headline cell. The \(\lambda = 0\) row inside each block is the control, and it is worth dwelling on: the simulated apparent gain there is not small, it is exactly zero, printed to ten decimal places, because with no noise the second measurement is the same floating-point number as the first for every individual. That row is the one place in the study where the answer is certain rather than estimated, and it is how we know the selection machinery is not manufacturing a gain by itself.

Checking It Against the Closed Form

A simulation that agrees with the formula it was written from proves nothing about the world. It proves the code implements the model. That is still worth establishing, and it is the only thing in this article that can be established with certainty, so here is how we did it.

The validation target is the shrinkage law rather than \(\Delta\) itself, because the law is the sharper statement. It says the selected group's mean on the second measurement, divided by its mean on the first, should equal \(r\) exactly, for every threshold and every noise level, and it should not care which threshold at all. So the ratio is a single number the simulation must hit 42 times from 42 different directions.

Three things had to hold, and we fixed what would count as a failure before running anything. First, no cell past the Bonferroni threshold. None was. Second, the pooled sum of squared \(z\) scores close to its degrees of freedom rather than above it, since a systematic bias too small to show in any single cell would still inflate the sum; 36.20 on 36 df is as close as one could reasonably ask. Third, the zero-noise control at \(r = 1\) returning exactly zero rather than something small, which it did at all six thresholds.

The randomised design gave us a fourth check for free. Across the 18 trial cells its estimate of the true treatment effect was never more than 1.77 standard errors from the truth, which is what an unbiased estimator looks like when you run it eighteen times.

Watching the number settle

Monte Carlo answers arrive slowly and from the wrong direction first. Figure 4 is the running estimate of the headline number as the selected individuals accumulate, from the first ten thousand to the last of six hundred thousand.

The trace is a useful thing for a club to look at, because it shows what a small run would have told us. Ten thousand selected people, which is more than most real studies have, gave 0.8518. That is a perfectly respectable answer and it is wrong in the second decimal place. The envelope tells you which digits to trust, and the honest way to report a Monte Carlo number is to report the digits inside it and no more.

Two Designs, the Same Simulated Patients

So far there has been no treatment. Now add one. A true effect \(\tau\), a constant shift in standard deviations, applied to the second measurement of anyone treated. Then run two designs on the same simulated people.

The single-arm design treats everybody who was selected and reports the mean change from before to after. The randomised design splits the selected group in half at random, treats one half, and reports the difference between the two halves on the second measurement. Both start from the same 600,000 selected individuals. Both have the same regression to the mean acting on them. One reports it as a treatment effect.

The left panel contains the entire argument of this article in one shape. The single-arm curves are parallel to the truth, sitting above it by a fixed amount. That fixed amount is \((1-r)\varphi(z_p)/p\), the same product from §3. The randomised points sit on the truth. Not approximately: across 18 cells the largest departure was 1.77 standard errors.

The right panel is where it bites. Because the bias is a constant and the true effect is not, the overstatement factor is worst exactly where the stakes are highest, which is for small real effects. In the headline scenario a true effect of 0.20 SD is reported as 1.077, an inflation of 5.38 times, with 81.5% of the reported result supplied by the selection. A true effect of 0.10 SD is inflated 9.77 times. A true effect of zero is reported as 0.877, and reaches \(p < 0.05\) in 100% of replicates.

That last number deserves its own sentence. A before-and-after design applied to a selected group does not have a 5% false-positive rate. On these settings it has a 100% false-positive rate, because the thing it is testing is not zero. The test is working perfectly. It is answering a question nobody meant to ask.

The difference-in-differences form of the randomised estimate, which compares change scores between arms rather than final values, was unbiased too, and in these cells was slightly noisier. Vickers and Altman set out when each is preferable [11]. The point for our purposes is that randomisation, and not the choice of summary, is what removes the bias: both arms regress by the same amount and the comparison subtracts it away.

Our Own Homework

We should say where this touches us, because it does.

The club runs sessions where members work through problems and then try the same kind of problem again later. If we picked out the members who did worst the first time, gave them help, and looked at how they did the second time, we would see them improve. We would then be tempted to conclude that our help worked.

We do not have paired quiz data. The club never collected any, and we are not going to pretend otherwise in an article about not fooling yourself. What we can do is run the model at class scale and see what it predicts. A class of 30, a short quiz with test-retest reliability 0.70, the bottom eight picked out, and an intervention that in the model does exactly nothing. Two hundred thousand simulated classes.

On a hundred-point scale with a mean of 68 and a standard deviation of 12, which is an ordinary-looking classroom spread, that is the bottom eight rising from 53.7 to 58.0 marks. Four point three marks. In 91.4% of simulated classes they improve. A teacher running this once, in one class, would see it work.

The second column is the one that changed how we think about it. Leave the top eight entirely alone and they fall by 4.29 marks, and they fall in 91.3% of classes. Same arithmetic, same size, opposite sign. Nobody writes an article about how doing nothing to the best students made them worse, but the model produces both halves from the same line of algebra, and if you believe the first you have already committed to the second. Meanwhile the class as a whole moved by 0.0007 standard deviations, which is to say it did not move.

Kahneman and Tversky made this point in 1973 with flight instructors who had noticed that praising a good landing was followed by a worse one and criticising a bad landing by a better one, and had drawn the obvious and wrong conclusion about praise [4]. Our class of 30 is the same shape. Any teacher, coach or manager who intervenes on the worst and then measures again is running this experiment, usually without a control group, always with a result.

The Strongest Objection We Could Make

The objection runs like this, and we think it is largely right.

Everything above compares two pieces of mathematics. The shrinkage law is a property of the bivariate normal, proved in the nineteenth century. We simulated a bivariate normal. It shrank. A student with a normal table could have written down every number in our grid table to within the error bars before we ran anything. That \(\sum z^2\) came out at 36.20 on 36 degrees of freedom tells you about numpy's normal sampler and about our indexing. It tells you nothing about clinical trials or classrooms.

We accept that. Three things survive it.

The first is that the objection applies to the validation, which is where it belongs, and we labelled the validation as a code check rather than dressing it as a finding. Its value is in what it would have caught. It did catch something, as it happens: the sign error in §4's notes was found because the printed prediction disagreed with the simulation.

The second is that the size is not obvious even when the mechanism is. Every member of the club knew what regression to the mean was before we started. Not one of us guessed 0.88 SD. The usual textbook presentation, including the two excellent BMJ notes by Bland and Altman [7][8], teaches the phenomenon with a small illustrative example and leaves the magnitude as an exercise. The product \((1-r)\varphi(z_p)/p\) is not hard, and it is also not something people carry around. Putting a number on it, across the range of thresholds and reliabilities that real studies actually use, is a modest contribution and it is ours.

The third is the comparison in §11 against a published figure, which we could not have written down in advance and which does say something about the literature rather than about our code.

A second objection is harder on us. Our model has no natural history. Real patients get better on their own; real students learn; real measurements drift. Krogsbøll and colleagues pooled 37 three-armed trials and found a standardised change from baseline of \(-0.24\) in the arms that received no treatment whatsoever [15], and that number contains real recovery as well as regression. We cannot separate them, and we do not claim to. What our model establishes is the arithmetic ceiling: how much of such a figure regression could supply by itself. Whether it did, in any particular trial, depends on that trial's entry rule and its instrument's reliability, which is exactly the pair of numbers that trial reports least often.

A third objection we have no answer to at all. Reliability \(r\) is handed to our model as a known constant. In practice it has to be estimated, often from the same small sample, and every correction for regression to the mean is only as good as that estimate [8][14]. Barnett and colleagues set out the correction properly [14]. We measured the disease. We did not measure the cure.

What This Does and Does Not Tell You

The finding is a product of two numbers, and both of them are chosen by whoever designs the study.

How extreme a group you select sets \(\varphi(z_p)/p\), which runs from 0.80 at the median to 2.67 at the worst percentile. How noisy your instrument is sets \(1 - r\). Multiply them and you have the improvement your study will report if the treatment does nothing. If that number is larger than the effect you are hoping to detect, a single-arm before-and-after design cannot tell you anything, however many participants you enrol, because more participants shrink the error bar around a biased estimate rather than the bias.

Our comparison against the literature works out as follows. The pooled change from baseline in the untreated arms of those 37 trials was \(-0.24\) [15]. If such a trial enrols the worst tenth on a single entry measurement, regression alone accounts for the whole of that figure as soon as the entry measurement's reliability is 0.863 or below. At the worst 1%, reliability 0.910 suffices. At a looser entry criterion of the worst half, reliability would have to fall to 0.699. Single-occasion clinical and behavioural measurements are routinely less reliable than 0.86, which is why blood pressure epidemiology developed an entire correction industry around the problem [6][9]. So the published \(-0.24\) puts no floor under how much of it is genuine recovery. It is consistent with all of it being selection, and consistent with none of it being selection, and a single-arm design has no way to tell. Note which direction the tight entry criterion pushes: the stricter your enrolment, the larger your artefact. A design choice made to sharpen a study makes its uncontrolled version less interpretable.

What the study says nothing about: any particular trial, any particular instrument, any real patient or student, whether a given published effect is real, or how to estimate \(r\) for anything. We did not look at a real dataset. We looked at a formula, hard.

Two practical readings we would defend. A before-and-after improvement in a group chosen for being extreme is not evidence of anything until somebody names \(p\) and \(r\); those two numbers, and a normal table, produce the null expectation in a line. And the fix is old and cheap. Randomise within the selected group and the bias cancels exactly, because both arms regress by the same amount, which is what Figure 5's open markers are showing. Yudkin and Stratton [10], Morton and Torgerson [13] and Barnett and colleagues [14] all say versions of this, and the field has known it since Davis in 1976 [5] and arguably since Hotelling told Secrist in 1933 that his long study of the decline of exceptional businesses had demonstrated nothing except that the data were correlated below unity [2][10]. The knowledge is not new. What we hope is new here is the size, printed where a reader can check it.

Where a different choice would change the answer

Normal against heavy-tailed. Of everything we assumed, this one moves the answer most. The factor \(\varphi(z_p)/p\) is a normal-tail quantity. Give the noise a heavier tail and the selected group is further out, the shrinkage law \(\mathbb{E}[X_2|X_1] = rX_1\) stops holding exactly because linear conditional expectation is a bivariate normal property, and the size of the artefact changes in a direction that depends on which distribution you pick. We did not vary it. Read every number in this article as conditional on normality, and expect a real noisy measurement with occasional wild readings to behave worse, not better.

A stable trait against a drifting one. We froze \(T\) between measurements. If the trait itself wanders, the correlation between occasions falls with the gap between them, so \(r\) becomes a function of elapsed time and the artefact grows with follow-up length. That is the regression dilution problem in its usual form, and it is why blood pressure epidemiology corrects associations for the attenuation caused by single baseline readings [6][9]. Our constant-\(r\) result is the baseline that departure starts from.

A fixed cut against order statistics. Section 9 shows the size of this: at \(n = 30\) the two differ by 0.010 SD, about 3%. At the sample sizes in the main grid they are indistinguishable. Anyone applying our numbers to a group of ten or twenty should use the order-statistic version, which is the one the interactive model runs.

Selection on one measurement against selection on an average of several. This is the cheapest fix available and we did not include it in the grid. Average \(m\) independent screening measurements before selecting and the effective reliability rises to \(mr/(1 + (m-1)r)\). At \(r = 0.5\), two screening visits take the effective reliability to 0.667 and cut the artefact by a third; three take it to 0.75 and halve it. A trial that screens twice before enrolling has already disposed of a third of its regression bias, and this is a design change, not an analysis one.

A constant treatment effect against one that varies with the trait. We added the same \(\tau\) to everybody. If the treatment helps the worst-off more, the single-arm estimate and the randomised estimate both change, and they change differently, which is the territory of Lord's paradox [3] and of the long argument about change scores [12]. Our clean separation between bias and effect depends on the effect being additive, and that is an assumption rather than a result.

Reproducing this

Everything in this article comes from one file and one command. You need Python 3.12 and numpy. No other package is used.

python regression-to-the-mean.py > regression-to-the-mean-output.txt

Expect about 40 seconds on a modern laptop. Ours took 36.0 s on numpy 2.4.2 and Python 3.12.3. The master seed is 20260211, hard-coded at the top of the file, and every grid cell, trial cell and auxiliary run draws its own independent stream from it through numpy's SeedSequence.spawn, so the whole output is deterministic and the order of the cells does not affect any of them. Peak memory is modest, set by the class simulation's 200,000 by 30 arrays. Numbers differing from ours by more than the printed standard errors mean something is wrong, and we would like to hear about it. The interactive model runs a smaller version of the same two experiments in your browser, with the selection threshold on a slider.

References