When the First Digit Test Works and When It Accuses the Innocent

A Test With a Reputation

Somebody hands you a spreadsheet. Ten thousand invoice amounts, or a county's vote tallies, or a decade of a lab's measurements. You want to know whether it was made up. There is a test for that, and it takes thirty seconds to run: look at the first digit of every number and see whether 1 shows up about 30% of the time, 2 about 18%, all the way down to 9 at under 5%. Real, unmanipulated collections of numbers tend to obey this distribution. Fabricated ones tend not to, because people inventing numbers reach for the middle digits far more evenly than nature or arithmetic ever does. The test has a name, Benford's law, a following among forensic accountants and election monitors, and, we think, a reputation somewhat larger than its warrant.

We are not the people to debunk it. The mathematics of Benford's law is genuinely elegant and we spend a fair portion of this article admiring it. What we distrust is a narrower thing: the leap from "this is a real mathematical regularity" to "therefore a dataset that fails it is suspicious." That leap needs a second number nobody usually reports, which is how often an honest dataset fails anyway. We built that number from scratch, several times, under several assumptions, and it is nowhere near as small as the confident tone of a fraud-detection slide deck implies.

What the Law Actually Says

The astronomer Simon Newcomb noticed it first, in 1881, from nothing more than which pages of a table of logarithms were worn soft by use: page 1 dirty, page 9 clean [1]. It sat unexplained and largely unread for 57 years until the physicist Frank Benford rediscovered it independently and tested it against 20,229 numbers from 20 wildly different sources, rivers, baseball statistics, atomic weights, street addresses, and published the frequencies that now carry his name [2]. Neither man had the scale-invariance proof that actually explains it; that came from Pinkham in 1961 [3] and was sharpened by Hill in the 1990s [4][5]. State it carefully once and the rest of the article can lean on it. For a positive quantity \(X\), write it as \(X = m \times 10^{k}\) with \(k\) an integer and \(m \in [1, 10)\) the mantissa. The leading digit is \(\lfloor m \rfloor\). Benford's law is the claim that \(\log_{10} m\), the fractional part of \(\log_{10} X\), is uniformly distributed on \([0, 1)\). Digit \(d\) occupies the sub-interval \([\log_{10} d, \log_{10}(d+1))\), whose width under a uniform mantissa is exactly

Two facts follow immediately and both matter later. First, the law says nothing about any specific quantity; it is a statement about the mantissa of a whole collection, so it can only apply to collections whose logarithms are smeared smoothly across whole decades, never to a single measurement. Second, it is scale-invariant: multiply every value in a Benford-conforming set by any positive constant and the digit law is unchanged, because multiplication just shifts \(\log_{10} X\) and a uniform distribution on a circle (which is what a periodic mantissa is) is invariant to shifts. This is why the law does not care whether your bank balances are in dollars or centavos. It is also the single cleanest reason population sizes, market capitalisations and river lengths tend toward it and human-chosen quotas, minimum wages and speed limits do not: nobody picked a preferred scale for the first kind of number, and everybody did for the second.

We checked our own arithmetic before trusting it on anyone else's numbers. The club's table of \(P(d)\) was computed two independent ways: once from the closed form above, and once by literally measuring, on a grid of 4,000,000 points, what fraction of \(u \in [0,1)\) satisfies \(\lfloor 10^u \rfloor = d\). The two agree to 5.3×10⁻⁸, which is the quadrature grid's own resolution and nothing more. The chi-squared statistic we wrote by hand from its definition, \(\chi^2 = \sum_d (O_d - np_d)^2 / (np_d)\), was checked against SciPy's chisquare on the bank-assets table below and the two statistics agreed to the last printed digit, difference +0.000×10⁰.

Eight Datasets, One Test

We picked eight series that share no author, no motive and mostly no unit, downloaded once on 2026-09-14 and cached under analysis/data/: country populations and life expectancies from the World Bank, CODATA 2022 physical constants from NIST, every FDIC-insured US bank's total assets and total deposits from a single quarterly Call Report filing, exoplanet orbital periods from NASA's Exoplanet Archive, and a full calendar year of USGS earthquake magnitudes above M3.5 together with the seismic moments those magnitudes encode. All eight downloads succeeded. Nothing here was substituted or invented; a dataset that failed to download would have been dropped and reported, and none did.

Plain Arithmetic

No simulation in this section. Just the sum a reader could redo with a calculator, because the country-population row above is the one place in this study where two standard verdicts disagree, and disagreements deserve to be shown rather than asserted.

217 countries and territories, 2023 World Bank population figures, first digit counted for each: 68 start with 1, 36 with 2, 30 with 3, 16 with 4, 25 with 5, 16 with 6, 8 with 7, 10 with 8, 8 with 9. Benford's expected share of 217 for each digit is \(217 \times P(d)\): 65.32, 38.21, 27.11, 21.03, 17.18, 14.53, 12.58, 11.10, 9.93. The chi-squared contribution of each digit is \((O-E)^2/E\):

Add the last column and you get 7.61, which the script also reports as 7.6 after its own rounding. Look that up against a chi-squared distribution on 8 degrees of freedom and the upper tail is 0.4726: nearly half of all genuinely Benford datasets of this size would show a chi-squared statistic at least this large just from sampling noise. By that test, country populations are unremarkable.

Now take the mean absolute deviation instead: \(\text{MAD} = \frac{1}{9}\sum_d |O_d/n - P(d)|\). Divide each observed count by 217, subtract Benford's proportion, take the absolute value, average the nine numbers, and you land on 0.01521. Nigrini's published band calls anything at or above 0.015 nonconformity. Two thousandths of a unit above the line, on a statistic with a bootstrapped 95% interval of [0.0121, 0.0329] on this very n, decides whether the verdict reads "close enough" or "flagged." Both numbers came out of the same 217 population figures. Neither is wrong. They simply answer different questions, one about how surprising the whole pattern is and one about how far any single average deviation has drifted, and a threshold applied to the second without reference to sample size is exactly the failure mode the rest of this article measures.

The Mechanism: Spread Is Everything

Why did bank assets pass and earthquake magnitudes fail so badly? The working hypothesis, stated in §2, is that Benford conformity is really a statement about how widely a dataset's logarithm is smeared across whole decades. We tested that directly rather than assuming it. Draw \(n = 5{,}000\) values with \(\log_{10} x\) uniform on a window of width \(S\) decades, at a random offset so the answer cannot be an artefact of where the window happens to start, and measure the resulting MAD. Repeat 400 times per \(S\) and also compute the exact answer analytically, since a log-uniform window's digit law is just the measure of each digit band intersected with the window.

The result is not what a rough intuition "wider is better" would predict. It is scalloped. At every whole number of decades, that is \(S = 1, 2, 3, \ldots\), a log-uniform window covers each mantissa exactly once, so the MAD is exactly zero, not approximately zero, by construction. Move away from a whole decade and the deviation grows, peaks somewhere past the half-decade mark, then shrinks back toward zero as the next whole decade approaches. The envelope of those peaks decays like \(|\sin(\pi S)/(\pi S)|\), a genuine sinc function, because the mantissa density of a width-\(S\) window is a Fourier series whose \(k\)-th harmonic carries a factor \(\sin(\pi k S)/(\pi k S)\). We checked this prediction rather than merely illustrating it: at every half-integer \(S\) from 0.5 to 9.5, the ratio of the analytic MAD to the sinc envelope should land on one fixed constant regardless of \(S\), because every even harmonic cancels there by symmetry. Across ten independent windows the ratio held to better than one part in a million.

D2, the physical constants, is the case that resists this story and deserves saying so plainly. It spans 114.34 decades, from Planck-scale quantities near 10⁻⁶⁵ to nuclear magnetic moments and inverse fine-structure figures at the other end, and it still fails nonconformity (MAD 0.0172). Spread is necessary but, on its own, not sufficient. CODATA's table is not 355 independent draws from a smooth distribution; many entries are exact algebraic multiples, reciprocals, or unit conversions of others (the Rydberg constant, the Rydberg constant times hc, the Rydberg constant times c, all three rows), so the effective sample size behind that spread is far smaller than 355, and the assumption of independence our chi-squared test leans on is simply false for this dataset. We say so in the script's own docstring rather than quietly and it is worth repeating here: a wide spread is what makes conformity possible. It does not by itself make conformity true.

For a mechanism this clean we wanted one more check with a known, computable answer rather than a simulated one: the leading digits of \(2^k\). Because \(\log_{10} 2\) is irrational, \(k \log_{10} 2 \bmod 1\) is equidistributed, a theorem rather than a tendency, so the first digit of \(2^k\) must converge to Benford's law exactly, and we can check our convergence against a right answer nobody needs to estimate. Naively computing this in double-precision floating point actually failed once in the first 2,000 terms: at \(k=3\), \(2^3 = 8\) exactly, but \(10^{3\log_{10}2}\) rounds to 7.999999999999998 in double precision and the floor takes 7 instead of 8. We fixed this by carrying \(\log_{10} 2\) to 50 decimal digits, split into a high part exact in double arithmetic and a low remainder, and confirmed all 2,000 terms then match exact integer arithmetic. At ten million terms the resulting MAD is 4.35×10⁻⁷, against roughly 7.4×10⁻⁵ for a random Benford sample of the same size, a genuine equidistribution converging roughly like \(1/N\) rather than the \(1/\sqrt{N}\) a random sample is stuck with.

Working Notes From the Club Table

How Often the Honest Get Accused

Here is the arithmetic a fraud-detection workshop rarely puts on a slide. Generate data that is exactly, perfectly Benford by construction (a multinomial draw from \(P(d)\) itself, so there is no fraud, no error, and nothing to find), and run it through the standard test. Do this 40,000 times at each of nine sample sizes.

The chi-squared columns are exactly what a correctly calibrated test should show: close to 5% and 1% at every n, largest deviation across all nine rows a mere 1.90 standard errors from the nominal rate. That is a genuine relief, and also, on reflection, not a very demanding thing to ask of a test, since the null distribution of a chi-squared statistic under a true null is chi-squared by definition. The interesting column is the last one. Nigrini's 0.015 line, applied without adjustment for n, flags 91.8% of perfectly honest size-100 samples and does not fall under 1% until n passes roughly a thousand. A statistic whose sampling noise shrinks like \(1/\sqrt{n}\) cannot be judged against a threshold with no n in it.

Real honest data is rarely exactly Benford; more often it is approximately lognormal, smeared over some finite number of decades \(\sigma\). Formann's survey of which common distributions approach Benford's law, and how quickly, makes the same point we keep landing on: it is the spread of the distribution relative to a decade, not its family name, that decides the outcome, and several familiar survival-time and lifetime distributions satisfy the law only in specific parameter ranges rather than always [16][17]. We computed the exact first-digit law of a lognormal with a given \(\sigma\) (in closed form, by summing the normal CDF across decade boundaries, not by simulation) and then measured how often 40,000 resampled datasets from that law get flagged.

Read a column and size costs innocence: honest data of fixed spread gets flagged more often simply because someone collected more of it, since a fixed-width departure from Benford becomes statistically undeniable at high enough n even though it was never fraud. Read a row and spread buys innocence back, but only past roughly \(\sigma = 0.5\). Below that, honest lognormal data is essentially always going to fail, at every sample size we tested. This is not a corner case. A great many real, non-fraudulent processes (annual rainfall at one station, a species' body lengths, exam scores rescaled to a log axis) live at \(\sigma\) well under 0.5.

Finally, we asked the question a real reviewer actually faces: not "would idealised lognormal data pass," but "if the book of business we are looking at is honestly what it appears to be, would a fresh honest sample of the same size get accused." We took the empirical digit law of D3 (bank assets, which passed) and of D5 (exoplanet periods, which failed chi-squared badly), corrected each for the upward bias that comes from feeding a noisy estimate back into itself (the correction is \(p_{\text{corr}} = P + t(\hat p - P)\) with \(t = \sqrt{1 - 8/\chi^2_{\text{obs}}}\), which sets the assumed true separation to its unbiased value), and resampled.

At the bank's actual size, the honest, bias-corrected accusation rate is 5.32%, almost exactly nominal: the banks were never in any real danger. At the exoplanet catalogue's actual size, it is 100.00%. An honest astronomical survey, resampled from its own true digit law with the bias correction applied in its favour, still fails essentially every time. Nothing is wrong with the catalogue. The chi-squared test, run at this sample size on this particular departure from Benford, was never going to say otherwise, honest or not.

Two Earthquakes, Opposite Verdicts

D6 and D7 are, event for event, the same 19,652 earthquakes. D6 is the magnitude the catalogue prints. D7 is the seismic moment that magnitude encodes, via the standard seismological relation \(\log_{10} M_0 = 1.5M + 9.1\), which is textbook physics and no claim of ours. The point is what the exponential transform does to conformity. Moment spans 7.95 decades, well clear of the sub-one-decade danger zone that sinks magnitude itself, and yet D7 still fails: MAD 0.0486, chi-squared 7,470.8, nonconformity.

The cause is not spread. It is precision. The catalogue reports magnitude to one decimal place. Push a number with 20 possible decimal endings (.0 through .9, times a scale factor with no bearing on the mantissa) through a deterministic linear-then- exponential transform and you cannot manufacture more than 20 distinct mantissa values no matter how wide the resulting range looks, because the transform is one-to-one on a countable input set. We counted directly: 115 distinct mantissa values across 19,652 converted moments, against 141 distinct magnitudes in the catalogue. A mantissa confined to 115 discrete points cannot be uniform on a continuum, and no amount of additional decades helps.

To confirm the mechanism rather than merely assert it, we put the rounding back: spread each reported magnitude uniformly across the 0.1-wide bin the catalogue rounded it into (the least committal way to restore what was thrown away) and recomputed. Before: MAD 0.0486, chi-squared 7,470.8. After: MAD 0.0189, chi-squared 647.4, an 11.5-fold drop. Still technically over Nigrini's line at this enormous n, but the overwhelming majority of the original failure evaporates the moment the rounding is undone. The energies released by these earthquakes were always close to Benford. The reporting convention, one decimal place, chosen for reasons having nothing to do with digit statistics, was not. A forensic reviewer handed only D7 and told nothing about its provenance would have no way to distinguish "this dataset was tampered with" from "this dataset was rounded before I received it," and the test alone supplies no such distinction.

The Strongest Objection We Could Make

The objection runs like this. Every quantitative claim in this article is a restatement of known mathematics. Hill proved the mantissa's uniformity is the unique scale-invariant law decades ago [4][5]. Pinkham proved the scale-invariance argument in 1961 [3], and Fewster later showed the whole result follows from an even simpler argument about averaging distributions of wildly different scale together [6]. That data confined to under a decade cannot be Benford is arithmetic, not discovery, once you accept the mantissa definition. Nigrini himself has written that his thresholds were calibrated on specific audit populations, not handed down as universal law [9]. So what, precisely, did simulating any of this teach anyone who already understood the definitions?

We think the objection is mostly right, and the parts of this study that survive it are narrower than the abstract might suggest.

The mechanism sections (§5, §8) are demonstrations of known results, and we say so rather than dressing a confirmation up as news. Their value is not the discovery but the discipline: showing that our simulated data actually does what the theorem says it should, and our earthquake dataset actually fails for the reason we claim rather than for some other reason we failed to notice, is a check on us, not a finding about the world. A different club could have written the same code, gotten a bug, and published a false claim with total sincerity. Running the check is what stands between those two outcomes.

What we would defend as a genuine, checkable contribution is narrower and more useful: the specific false-accusation numbers in §7, computed for realistic sample sizes using this study's own datasets rather than idealised toy examples. That honest exoplanet data resampled at its own real n of 6,016 fails 100% of the time even after a bias correction that favours innocence is not implied by "the mantissa definition." It required actually building the resampling machinery, actually computing the bias correction Judge and Schechter's methodology implies [8], and actually running it on a real astronomical catalogue rather than a synthetic one. Diekmann made a related point about scientific data specifically [7], and our exoplanet result is a companion data point to his, not a replacement for it.

A second version of the objection cuts deeper and we have no full answer to it. Every honest-data model in this article, multinomial-Benford, lognormal, log-uniform, and empirical-law-plus-bias-correction, is still a model we chose. Real invoices carry round-number spikes, reporting floors, currency conversion artefacts and human rounding habits that none of our five null models fully captures. We tested one such habit (rounding to a fixed number of significant figures, §7) and found the test blind to it. We did not test rounding to a fixed currency unit, which is the far more common real habit and, we suspect, would move the false-accusation rate further still. That experiment is future work, not a result we can report today.

Where a Different Choice Would Change the Answer

Several decisions in this study could have gone otherwise, and saying which ones would have moved the headline numbers seems the least we owe a reader who wants to reuse any of this.

Chi-squared against a G-test or Kolmogorov–Smirnov. We used Pearson's chi-squared throughout because it is what Nigrini's own literature and most published election-fraud applications use [10][11][13], which makes our numbers directly comparable to that literature. A log-likelihood-ratio G-test would give a numerically close but not identical statistic on small samples; a KS test on the empirical mantissa CDF would be more sensitive to a smooth, uniform shift and less sensitive to the specific digit-by-digit spikes a chi-squared test targets. The false-accusation qualitative story (honest data of narrow spread gets flagged, honest data of wide spread mostly does not) would survive the swap. The exact percentages in §7 would not.

First-digit against first-two-digit. Nigrini's own applied work leans heavily on the first-two-digit test precisely because it is more sensitive [9]. A first-two-digit version of this study would very likely convict D3 and D4 of nothing while still catching D6, but we cannot report that number, because we did not run it: this study is scoped to the first digit only, stated as a limitation in the script's own docstring.

Nigrini's MAD bands against a size-adjusted MAD threshold. The entire point of §7 is that a fixed 0.015 cutoff misbehaves across n. Several authors have proposed n-adjusted alternatives; we did not adopt one, because the fixed threshold is what real-world forensic software and several published applications actually deploy [12][14], and measuring the tool as used, warts included, was the point of this study rather than measuring an improved tool nobody uses.

Which countries count. D1 excludes World Bank aggregate rows (regions, income groups) using the Bank's own metadata endpoint. Leaving those aggregates in would inflate n with numbers that are sums of other numbers already in the table, which is close to the textbook definition of violating independence, and would very likely move D1's verdict. We checked that we excluded them; we did not explore what including them would have done, because doing so on purpose would have been the wrong analysis to report.

What We Would Not Accuse Anyone Of

Start from the number that started this article: an honest dataset of size 100, exactly Benford by construction, fails Nigrini's standard MAD test 91.8% of the time. Whatever else this study found, that number alone should retire any claim that a failed first-digit test is, by itself, evidence of anything, for a dataset anywhere near that size. Move to a few thousand records and the naive rule calms down, but the underlying generating process still has to actually be close to Benford, which §5 and §9 show is a much narrower requirement than "the numbers are honest."

None of this makes Benford's law useless as an investigative tool, and we want to be precise about the distinction, because the sceptical tone of this article is aimed at one particular use and not at the mathematics. As a lead generator among many transactions of comparable kind and comparable scale, where an investigator already expects Benford conformity from the population being screened and is looking for the specific accounts that stand out from their peers rather than treating a single aggregate statistic as a verdict, the test has a defensible track record [8][9][10]. De Marchi and Hamilton used exactly this framing on self-reported industrial pollution data, flagging specific facilities for follow-up rather than convicting an entire national inventory [15], which is the shape of application these results actually support. As a courtroom-ready proof that a specific dataset was fabricated, on the strength of a chi-squared p-value or a MAD threshold alone, without also reporting how often the same test convicts honest data of the same size and shape, it is not supported by anything in this study, and we could not find support for it in the literature we read either.

We came to this admiring the mathematics more than when we started, not less. A single formula, \(\log_{10}(1+1/d)\), derived from nothing more than "the answer should not depend on your choice of units," correctly predicts the digit statistics of bank balance sheets, planetary orbits and the constants of physics, and fails exactly where the derivation says it must, on data crammed into less than a single order of magnitude. That the same formula gets marched into courtrooms and audit reports as a stand-alone verdict, with its own false-accusation rate left unmeasured, is not a flaw in the mathematics. It is a flaw in how the mathematics gets used, and it is the only part of this we set out to be sceptical about.

Reproducing this

Everything in this article comes from one file and one command. You need Python 3.12 and numpy; SciPy is used for one cross-check if importable and the script runs correctly without it.

python benford-real-data.py > benford-real-data-output.txt

Expect about 20 to 40 seconds on a laptop once the eight datasets are cached under analysis/data/ (roughly 15 additional seconds and 6 MB on the very first run, downloading from the World Bank, NIST, the FDIC, NASA and the USGS). Our run took 21.1 seconds on numpy 2.4.2, SciPy 1.13.1 and Python 3.12.3. The master seed is 20260914, hard-coded at the top of the file, with every simulated section spawning an independent stream from it, so the output is deterministic byte for byte. A rerun that disagrees with the archived output by more than sampling noise, or with the retrieval date's cached files replaced by a fresh download, means something changed and we would like to hear about it. The interactive model runs a smaller version of the same tests against the same cached datasets in your browser.

References