INTERACTIVE COMPANION · META-ANALYSIS · ECOLOGY

The Pooling Bench

A live model accompanying “The Insect Decline Literature Disagrees With Itself”

Two benches. The first lets you throw studies out of a meta-analysis by clicking them, and watch the pooled estimate, its interval and the heterogeneity statistic move while you do it. The second lets you manufacture an insect decline out of a world where nothing is declining, by choosing monitoring sites the way real monitoring programmes were chosen.

Everything here matches the article's Python line for line, including the random number generator, so the defaults print the article's numbers exactly rather than approximately.

Bench 1. Eleven studies, one pooled number

Each row is a published estimate of the annual rate of change in total insect abundance or biomass, with the standard error the source paper printed. The diamond at the bottom is the DerSimonian-Laird random-effects pool. Click any row to drop it from the pool or put it back.

The quantity to watch is not the diamond. It is \(I^2\), the share of the spread between studies that sampling error cannot explain. Convention calls anything above 75 percent considerable. Try to get it below 85 by removing studies you dislike, and notice how much of the drawer you have to empty.

Preset selection These are the sensitivity analyses from §9 of the article. Start with “one effect per paper”: the pool gets steeper, not shallower, which is the opposite of what most people guess when you tell them a meta-analysis is double-counting. Model random effects Switch to fixed effect and watch the diamond jump to −4.26 percent per year and the interval shrink to almost nothing. That number is 53 percent one German study. This control exists so you can see exactly how a defensible model choice moves a headline. Interval level 95% Drag it to 99 and the pooled interval still excludes zero. Drag it to 99 with only four or five studies left in and it usually does not. Show prediction interval The dashed bar. It answers “what will the next study find” rather than “what is the average”, and at the default settings it runs from −8.14 to +3.10 percent per year. It crosses zero. That is the honest summary of this literature.

in pool: – pooled: – interval: – I²: – τ: – Q: –

–

What the weights are doing

Under a fixed-effect model the weight of study \(i\) is \(1/\mathrm{SE}_i^2\), so a study with a standard error of 0.002 gets 25 times the say of one at 0.010. Under random effects the weight becomes \(1/(\mathrm{SE}_i^2 + \hat\tau^2)\), and when \(\hat\tau^2\) is large compared with every \(\mathrm{SE}_i^2\), the denominators become nearly equal and so do the weights.

That is why the boxes in the plot are all almost the same size. It is also why removing the most precise study barely moves the diamond while removing the only positive study moves it quite a lot. Precision stopped buying influence some time ago; position still buys it.

Bench 2. Making a decline out of nothing

Here is the strongest argument against the whole literature, made concrete.

Nobody puts a trap where there are no insects. Reserves get designated because they are notably good, volunteers start transects on meadows full of butterflies, and field stations get built where there is something to study. Selection on abundance is built into how monitoring comes to exist, and it acts on the observed abundance in the year of selection, which contains a year-specific fluctuation sitting on top of the site's permanent quality.

The simulation below builds a world with four hundred sites. Each site has a permanent quality drawn from a normal distribution, plus independent year-to-year noise. Then it keeps only the sites whose first year looked best, fits a straight line through each survivor on a log scale, and averages the slopes. Set the true trend to zero and a decline appears anyway. That is regression to the mean and it is not a bug in anybody's statistics.

Sites kept, ranked on year one 10% 100 percent is no selection at all and the apparent trend goes to zero, as it should. Slide left and the manufactured decline grows monotonically. Even at 2 percent, an absurd degree of selection, it does not reach the pooled estimate from Bench 1. Year-to-year noise SD 0.60 log units 0.6 means a good year catches about 1.8 times a bad one at the same site. Push it to 0 and the whole effect vanishes, which is the cleanest demonstration that the mechanism really is regression to the mean and nothing else. True underlying trend 0.00 %/yr Set this to −1 and watch the observed trend come out steeper than −1. Selection does not only invent declines from nothing, it inflates real ones, which is the reason to read published rates as upper bounds. Length of series 27 years 27 matches the Krefeld window. Short series are hit much harder, because the fluke first year is a larger share of the line being fitted. Drag to 8 and see. Keep resampling Off uses the same 24 seeds as the article's Python, so the readout below matches the printed table. On rolls fresh seeds so you can watch how much the answer wanders, which is worth doing once before trusting any single simulation of anything.

sites monitored: – apparent trend: – over the whole series: – manufactured by selection: – share of our pooled estimate: –

–

Default settings: top 10% of sites kept, noise SD 0.60, true trend 0.00 %/yr, 27 years
Apparent trend: −0.39 %/yr, which is −10.1% over the whole 27 years
Share explained: 14.6% of the article's pooled estimate of −2.68 %/yr
Harshest setting: top 2.5% kept, noise SD 1.0: −1.36 %/yr, about half the pooled estimate
Verdict: real, quantified, and not large enough to be the whole story

What would change our minds

Four things, and none of them are things a school club can do.

Raw catch data from the major monitoring programmes, run through one common analysis pipeline with one agreed treatment of the reference year, would collapse a large share of our \(I^2\) into something interpretable. Someone would then be able to say how much of the 96.3 percent is genuine spatial difference and how much is method. Nobody can say that now, including us.

Monitoring programmes that record why their sites were chosen would let the selection-bias calculation in Bench 2 be done with real numbers rather than plausible ones. At the moment that entire argument, in both directions, is being conducted with assumed parameters.

A pre-registered systematic review with two independent screeners would replace our convenience sample. If it found a pooled estimate near −1 percent per year, we would believe it over ours without hesitation, because its search procedure would be better than ours by every measure that matters.

And more positive results in print. The Rothamsted suction trap that shows insects increasing is one of eleven here. If the true proportion of increasing sites is anything like what our \(\hat\tau\) implies, roughly a sixth, then the published record is missing them, and that missing sixth is doing more damage to the literature than any single overstated headline.