Counting Every Word in Fifty Books to Test a Law Nobody Can Explain

A Law That Should Not Survive Contact With Real Books

Here is the claim, stated the way it gets repeated at parties. Take any long piece of writing. Count every word. Rank them by how often they appear. The most common word will show up about twice as often as the second most common, three times as often as the third, four times as often as the fourth, and so on, in a curve so regular it has a name and an exponent. George Kingsley Zipf noticed it in the 1930s and spent a career arguing it was a law of human behaviour rather than an accident of English [1]. Nobody has ever produced a first-principles derivation of it that survives contact with everyone else's objections; Piantadosi's widely cited review of the whole literature reaches, after surveying decades of competing accounts, more or less that same conclusion [6]. Ninety years on, the exponent is still measured empirically, book by book, language by language, and the mechanism behind it is still, honestly, disputed.

That is an unusual thing to be able to say about a law this famous, and it is the reason we picked this topic. We had computers, fifty books nobody needed a lab to obtain, and an afternoon that turned into several. We expected a tidy confirmation. We got a confirmation that is real, an exponent that is not exactly what the textbooks claim, and at least one place where the whole picture visibly comes apart in the tail of rare words. All of that is more interesting than the tidy version would have been.

What follows states the law precisely, describes the fifty books and how we checked our own arithmetic before trusting it on them, walks one calculation by hand, reports where the law holds and where it visibly does not, sets our numbers beside two other people's published ones including a disagreement we do not explain away, and closes with the single strongest argument against taking any of this seriously.

What the Law Actually Claims

Write the count of the word at frequency-rank \(r\) as proportional to \(r^{-a}\). That is the pure form. It is a poor description right at both ends of most real vocabularies, which is why Mandelbrot proposed a one-parameter generalisation, adding a shift \(q\) to the rank before it is raised to a power [2]:

We fit both forms two different ways, on purpose. On the rank side, a book of \(N\) tokens is treated as \(N\) independent categorical draws over its \(V\) distinct word types, ranked by observed count, and \(a\) (and \(q\), for Mandelbrot) is chosen to maximise the exact multinomial log-likelihood, a numerical search over the whole vocabulary rather than a straight line drawn through logged axes. On the frequency side, the count \(x\) of each word type is modelled directly as a draw from a discrete power law \(p(x) \propto x^{-b}\) for \(x \ge x_{\min}\), fit by the method of Clauset, Shalizi and Newman: maximum likelihood for \(b\) at every candidate \(x_{\min}\), the actual \(x_{\min}\) chosen by minimising the Kolmogorov–Smirnov distance to the data (a discrepancy measure Kolmogorov introduced in 1933 for exactly this purpose, comparing an empirical distribution to a candidate model [16]), and a goodness-of-fit \(p\)-value from a parametric bootstrap that reselects \(x_{\min}\) on every synthetic replicate [3]. The two exponents are related by \(a = 1/(b-1)\) when both descriptions genuinely hold, which, spoiler, they mostly do but not exactly.

Nowhere in either fit does a straight line get drawn through a log-log scatterplot. The textbook method for finding a Zipf exponent, ordinary least squares on \(\log(\mathrm{count})\) against \(\log(\mathrm{rank})\), is known to bias the estimate and to report standard errors from a formula (built for independent, homoscedastic residuals) that has nothing to do with the actual sampling process, a point Newman's widely cited review of power laws makes at length [7]. We do not use it anywhere in this study.

Does the exponent mean anything, in the sense of pointing to a mechanism? Nobody agrees. Kanter and Kessler showed that a simple Markov process, letters generated with realistic transition probabilities but no notion of meaning at all, produces word-length and rank statistics that look strikingly Zipf-like [13], and Li showed the same for text typed by hitting keys at random on a keyboard with a space bar, provided the space bar is disproportionately likely [10]. Both results are sometimes summarised as "Zipf's law proves nothing about language," which would make an entire subfield's central regularity an artefact of how rank-ordering statistics work on any long, discrete sequence. Ferrer-i-Cancho and Elvevåg pushed back directly: random-typing text and real language share the coarse shape of the curve but differ measurably in exactly the quantities this article spends the most time on, the behaviour of the exponent across rank windows and the size of the vocabulary relative to the text length [11]. We are not in a position to referee that argument. We can say that nothing in our own fifty-book measurement rules either side out: the law holds well enough to be worth measuring carefully, and it holds unevenly enough, as §7 through §9 show, that "it's just what random sequences do" remains a live hypothesis rather than a settled dismissal.

Fifty Books, Fourteen Languages, One Long Argument

All fifty texts came from Project Gutenberg [17], plain UTF-8, retrieved 2026-09-14 from https://www.gutenberg.org/cache/epub/<ID>/pg<ID>.txt, cached under analysis/data/gutenberg/ with a manifest recording every title, author, Gutenberg ID, language, release date, byte count and SHA-256 checksum. All fifty downloads succeeded; nothing was substituted. Nineteen are in English, spanning Hamlet (1601) to The Great Gatsby (1925). The rest are eight in French, five in German, three in Spanish, two each in Italian, Portuguese, Dutch and Latin, and one each in Finnish, Swedish, Danish, Hungarian, Greek and Esperanto, the last two being translations and labelled as such rather than counted toward their language of origin. The oldest text is Sophocles' Oedipus Tyrannus (a Greek text; the specific Gutenberg edition is a modern Greek rendering), first performed around 429 BCE. The newest is Fitzgerald's 1925 novel. That is roughly 2,350 years of writing, in fourteen languages, tokenised by exactly the same code.

A word here means a maximal run of Unicode letters with optional internal apostrophes, lowercased. Numerals, punctuation and markup are discarded, and Project Gutenberg's own boilerplate is stripped between its standard start and end markers. That is one modelling choice among several defensible ones, and §11 measures what three other choices do to the answer. Before believing any of the fifty vocabulary counts, we ran a crude language gate: the share of a text's tokens that are common English function words. Every English book scored between 14% and 21%. Every non-English book scored under half a percent, with the sole partial exception of the two Dutch texts at 0.45%, still far below the English floor. No file had leaked English front matter into its counted text.

Figure 1 plots the raw rank-frequency curve, counted directly, no fitting yet, for six of the fifty books: an English whale-hunting novel, an English novel of manners, a French historical novel, a Spanish picaresque epic, a Finnish rural saga and a medieval Italian poem, spanning the 14th to the 19th century. On log-log axes, a pure Zipf law is a straight line. All six curves lean toward one, and none of them is exactly one, which is the whole tension this article investigates in detail.

Call me Ishmael. Some years ago—never mind how long precisely—having little or no money in my purse, and nothing particular to interest me on shore, I thought I would sail about a little and see the watery part of the world. Moby-Dick, Herman Melville, 1851, opening lines, tokenised as 45 words, none of them repeated inside this quotation

Moby-Dick is the single most heavily studied word-frequency data set in the whole power-law literature, which is exactly why it appears twice below: once as one of fifty equal books, and once as the direct point of comparison with a published fit.

Checking the Machine Before Trusting It

Five validations ran before any book was opened, because a bug in the likelihood code would produce fifty confident, wrong, internally-consistent numbers and nothing about fifty books would ever reveal that.

A. The Hurwitz zeta function. The entire likelihood rests on \(\zeta(s,q)=\sum_{k=0}^{\infty}(q+k)^{-s}\), computed here by Euler–Maclaurin summation with sixteen direct terms and four Bernoulli corrections, no SciPy. Checked against seven closed forms, including \(\zeta(2,1)=\pi^2/6\) and Apéry's constant: worst relative error 1.35×10⁻¹⁵, machine precision.

B. The sampler. Two million draws from a known discrete power law were checked against the model's own exact CDF (KS distance 0.000646 against a critical value of 0.000960) and against an analytic mean (−0.63 SE). Both pass.

C. Recovering a known exponent. Synthetic data drawn from four known values of \(b\) (1.60 to 3.00) at five sample sizes (250 to 64,000), 300 replicates per cell: the largest bias-to-SE ratio across all twenty cells was 2.28, consistent with Monte Carlo noise rather than a real bias, and the formal standard error tracked the observed scatter at every cell, shown for one of the four true exponents in Figure 2.

D. Is the goodness-of-fit test itself calibrated? Eighty synthetic vocabularies generated from a true power law, put through the whole fitting-and-bootstrap pipeline including the \(x_{\min}\) search: rejection rate at \(p<0.05\) came out 6.2% against a nominal 5% (\(z=+0.51\)), and mean \(p\) was 0.505, both consistent with a correctly calibrated test. The same pipeline correctly rejected log-normal look-alike data 62% of the time.

E. The price of ranking your own data. A rank-side fit sorts types by their own observed counts, and that sorting is itself a source of noise once fed back into the fit. Synthetic Zipf–Mandelbrot books at known \(a\) and \(q\) show the bias runs the opposite direction from the naive guess: it pushes the fitted \(a\) down, not up, by roughly 1% at 200,000 tokens and by up to 9.5% at 20,000 tokens. Every exponent quoted below for a real book carries a bias of about this size and this sign. Comparisons between books survive it far better than any single value does, because the bias is close to constant across books of comparable length.

One of these validations caught a real bug. The first version of the discrete Kolmogorov–Smirnov statistic compared the empirical CDF just below an observed value against the model CDF at that value, instead of the model CDF just below it. At \(x=x_{\min}\) this manufactured a discrepancy equal to the entire first bin's probability mass. Validation B caught it immediately: \(D=0.127\) against a critical value of \(0.001\), on data drawn from the exact model being tested, which should never happen. Every number in this article comes from the corrected code.

Plain Arithmetic

No fitting in this section. Just the four biggest numbers in Moby-Dick, and the ratios anyone can check with a calculator, because the "twice, then three times, then four times" version of Zipf's law is precise enough to fail loudly, and it does.

The most common word in Moby-Dick, once boilerplate is stripped and everything is lowercased, is "the," which occurs 14,535 times across 216,603 tokens. The second most common word occurs 6,624 times. The third, 6,447. The fourth, 4,739.

If the pure law held exactly, "rank times count" would be the same constant on every row. It is not: 14,535, then 13,248, a 9% dip, then 19,341, a 46% jump back up, then 18,956, roughly holding. Rank 3 is the interesting failure. Zipf predicts a word occurring a third as often as "the," which would be 4,845. The actual count is 6,447, a third of the way back toward rank 2's value, because in real English "of" and "and" and "the" are so close in raw frequency that their relative order is almost a coin flip from one book to the next, and no smooth power law can be close to three separated integers that are, in reality, nearly tied. \(a=0.9983\) is the maximum-likelihood answer for the whole 17,371-word vocabulary of this book, a very good fit in aggregate (Figure 4 shows exactly how good), and it is simultaneously true that the law visibly does not nail the exact spacing of its own top four words. Both statements are correct at once. That is most of what this article is actually about.

Working Notes From the Club Table

Where the Straight Line Bends

A single exponent implies the same power law everywhere in the ranking, from the most common word to the rarest. We tested that by fitting a pure Zipf law separately inside four rank windows of the same book: ranks 1–100, 101–1,000, 1,001–10,000, and beyond 10,000 for the books with enough vocabulary to have a window there. If one exponent described the whole curve, the four numbers would agree.

They do not. Averaged across all fifty books, the head of the ranking (1–100) fits a shallower slope, mean 0.774 (SD 0.067), than the middle of the ranking (101–1,000 and 1,001–10,000), which average 1.066 and 1.050. The shift from head to upper-middle is consistent in direction across the corpus: it goes the same way in 41 of 50 books, paired \(t=+6.65\). Past rank 10,000, in the eighteen books with a large enough vocabulary to have such a tail at all, the fitted local exponent collapses toward 0.27 (SD 0.19) essentially because there is almost no data left to constrain it: most words out there occur once, and a handful of counts cannot pin down a slope.

Ferrer-i-Cancho and Solé reported something in the same shape from a single large corpus: two regimes, a shallower head and a markedly steeper tail beyond a crossover somewhere in the low thousands of ranks [5]. Averaging our own fifty books into the same two windows they used, ranks 1–100 against ranks 1,001–10,000, gives mean \(a=0.774\) against mean \(a=1.050\), the same direction and roughly the same size shift they reported from different data entirely. We did not set out to reproduce their finding; it fell out of a check built for a different reason, which is the kind of agreement worth taking seriously.

Rare words carry a second failure that has nothing to do with slope. Under either fitted model, the expected number of word types occurring exactly once (called hapax legomena) can be computed directly from the fit. In every one of the fifty books, both models underpredict it, and by almost the same fraction each time: the Zipf–Mandelbrot prediction averaged 45.8% of the true hapax count (SD 3.0 percentage points), under-predicting in 50 of 50 books. Real books are fuller of once-used words, proper nouns, foreign phrases, a single invented compound, than either smooth model expects. This is not a small-sample artefact confined to short books: it holds as strongly in Ulysses (16,195 observed hapaxes against 7,523 predicted) as in Hamlet (2,998 against 1,294). Models built specifically to track how a vocabulary fills in as more text is added, rather than treating the whole book as one static count, handle this kind of incompleteness far better than a single rank-frequency snapshot ever can [8][9][12]; a full large-number-of-rare-events treatment of the kind Baayen catalogues [14] was outside what this study set out to do, and the 45.8% figure above is the size of the gap such a treatment would need to close.

One Extra Parameter, Fourteen Percent

Zipf–Mandelbrot nests pure Zipf at \(q=0\), so it can never fit worse, and the question is whether the extra flexibility earns its keep, measured here by Akaike's information criterion, twice the number of parameters minus twice the log-likelihood [15]. Twice the log-likelihood improvement is also a chi-square statistic on 1 degree of freedom; the critical value at \(p=0.05\) is 3.84. Every one of the fifty books cleared it, with a median likelihood-ratio statistic of 4,166 and a minimum of 7.3, so even the weakest case clears the bar comfortably. Zipf–Mandelbrot won on AIC in 50 of 50 books.

The fitted shift \(q\) has a median of 3.03 across the corpus and ranges from −0.31 to 9.42, and it drags the exponent along with it: mean \(a\) under pure Zipf is 0.9549; mean \(a\) under Zipf–Mandelbrot is 1.0866, a shift of +0.1316, about 13.8% of the pure-Zipf value. Anyone quoting a Zipf exponent to three decimal places without saying which of the two forms produced it is quoting a number that moves by more than a tenth depending on a choice the reader was never told about.

Does It Actually Fit? Asking Honestly

Before the goodness-of-fit numbers, the question this whole study was built to answer: does the exponent hold across centuries? Figure 5 plots every one of the fifty books' fitted rank exponents against the year the work was first published, an axis broken between antiquity and the 14th century since nothing in the corpus falls between Sophocles and Dante, coloured by era.

The Clauset–Shalizi–Newman rule rejects a pure power law as an adequate description whenever the bootstrap \(p\)-value is at or below 0.10. By that rule, a pure power law is not ruled out for 34 of the 50 books, and the median \(p\) across all fifty is 0.138. That means a clear majority survive a real statistical test, not just a plot that looks straight-ish, and it also means sixteen books, a meaningful minority, do not. The frequency-side exponent \(b\) averaged 1.951 (SD 0.117) over all fifty books and 1.960 (SD 0.118) restricted to the thirty-four survivors, which is close enough that surviving the test is not simply a matter of which books happen to have a smaller \(b\).

The two exponents, rank-side \(a\) and frequency-side \(b\), should relate by \(a=1/(b-1)\) if both descriptions of the same book are correct. They come close: mean \(1/(b-1)\) is 1.0645 against mean rank-side \(a\) of 0.9549, a difference of \(+0.1096\) (SD 0.0761), correlated at \(r=0.790\) across the fifty books. They are not measuring identical things. The rank fit uses every word type in the book; the frequency fit, by construction, only uses the roughly half of types that sit above the chosen \(x_{\min}\). Agreement to within about a tenth, correlated three-quarters of the way to perfectly, across two methods that see different slices of the same vocabulary, is a real corroboration. It is not the same as the two numbers being interchangeable.

Two comparisons against other people's published numbers, one that agrees and one that does not. First, Moby-Dick against Clauset, Shalizi and Newman's own Table 6.1: they report \(x_{\min}=7\pm2\), \(b=1.95\pm0.02\); we independently found \(x_{\min}=8\), \(b=1.958\pm0.018\). Difference: +0.008, or 0.29 combined standard errors. Agreement. Holding our own fit at their \(x_{\min}=7\) instead of our preferred 8 gives \(b=1.9418\pm0.0169\) against their \(1.95\pm0.02\), a difference of −0.31 combined standard errors. Agreement either way the comparison is framed, on the single most heavily re-analysed book in the field.

Second, Moreno-Sánchez, Font-Clos and Corral fit the same Clauset–Shalizi–Newman machinery to 31,075 English Gutenberg texts and report a mean frequency-side exponent among the texts their test did not reject of about 2.03 (SD 0.15) [4]. Our nineteen-book English subset gives mean \(b=1.898\) (SD 0.061). Difference: −0.132, which is −9.47 standard errors of our own mean, not of theirs. That is not sampling noise. We do not think it is a bug either, and §10 explains why, but we are not going to round it away by only quoting the comparison that agrees.

The Strongest Objection We Could Make

The objection runs like this. Fitting a strictly more flexible model and finding it wins on every single book is a mathematical guarantee dressed up as a discovery: Zipf–Mandelbrot contains pure Zipf as the special case \(q=0\), so its maximised likelihood can never be lower, and AIC's 2-point penalty for the extra parameter is a fixed toll that a sufficiently long book's likelihood surface will always pay if the true curve bends at all near the origin, which real word-frequency curves always do to some degree. Fifty wins out of fifty proves less than it sounds like it proves. And the −9.47σ disagreement with a peer-reviewed 31,075-text survey is, on this reading, not a genuine scientific finding about how English differs from itself. It is what happens when a club with fifty canonical novels compares itself to a study built from the entire un-curated tail of Gutenberg, meaning fan fiction, government documents, cookbooks, and short pamphlets we deliberately excluded, run through a tokeniser neither study has published in full detail. Two studies that never counted the same objects will not report the same number, and calling the gap between them "−9.47 standard errors" dresses up an apples-to-oranges comparison in the language of a precise result.

We think the first half of the objection is correct and the second half is only partly correct.

On Zipf–Mandelbrot: yes, 50 of 50 wins on AIC is close to guaranteed by nesting, and we say so plainly rather than presenting it as a surprise. What the objection does not remove is the likelihood-ratio test's actual size: a chi-square statistic with a median of 4,166 against a critical value of 3.84 is not a model squeaking past a penalty, it is a model annihilating the comparison, and the smallest of the fifty statistics, 7.3, still clears the bar by a factor of two. The genuinely informative number in §8 was never "who won." It is the 13.8% average movement in the fitted exponent between the two forms, which is a real, checkable fact about how sensitive this whole literature's headline number is to an unstated modelling choice, and that number survives the objection intact.

On the disagreement with Moreno-Sánchez et al: we tested the "different tokeniser, different corpus" explanation directly rather than asserting it, which is the one thing that keeps this from being a shrug. §11 reruns our own nineteen English books under three other tokenisation rules. Rule D alone, keeping case rather than lowercasing, moves the rank exponent by a mean of −0.0151 across the corpus, with a largest single-book shift of 0.0428. On the frequency side, our own choice of edition, canonical novels only, no pamphlets, no pagination artefacts, no OCR noise, plausibly shifts the typical \(x_{\min}\) and therefore \(b\) by a comparable amount in the other direction, small books at the tidy end of literature tending to show a smaller \(b\) than a corpus that includes everything Gutenberg happens to host. We have not run Moreno-Sánchez et al.'s exact 31,075-text pipeline ourselves, so we cannot certify that tokeniser and corpus differences account for the entire 9.47σ gap rather than part of it. What we can say is that a difference of this general size and direction is fully consistent with modelling choices we can measure moving our own numbers by a similar amount, which is a considerably weaker and more honest claim than either "the two studies agree" or "one of us made an error."

Where a Different Choice Would Change the Answer

Several choices in this study could have gone otherwise, and saying which ones would have moved the headline numbers is the least we owe a reader who wants to reuse any of this.

Tokeniser. We compared our default rule (letters with internal apostrophes, lowercased) against three others on all fifty books: letters only, so "don't" splits into "don" and "t"; letters with internal hyphens also kept whole; and case preserved, so "The" and "the" count as different types. The first two move the mean exponent by less than the corpus's own half-to-half measurement noise (+0.0039 and +0.0001 respectively, against a noise floor of 0.0044). Keeping case does not: mean shift −0.0151, more than three times the noise floor, because capitalised proper nouns and sentence-initial words get split away from their lowercase twins and the vocabulary inflates. This is the choice that would have changed our headline number the most, and we made it deliberately, in the direction every comparable published study also makes it.

Word forms against lemmas. "Walk," "walks" and "walked" are three separate types in this study. A lemmatised count would merge them, shrinking the vocabulary and, almost certainly, steepening the fitted exponent, especially for the morphologically richer languages in this corpus (Finnish and Hungarian have far more inflected forms per word than English or the Romance languages do). We did not lemmatise, in any of fourteen languages, because a single defensible lemmatiser for all fourteen does not exist without external dependencies this study chose not to rely on. This is very likely the single largest unmeasured source of the between-language spread visible in §9's table.

\(x_{\min}\) by minimum KS distance against a fixed cutoff. The Clauset method searches for the \(x_{\min}\) that minimises the discrepancy between the model and the tail of the data. A study that instead fixed \(x_{\min}=1\) for every book, fitting the power law to the entire frequency distribution including the most common words, would report a systematically different \(b\), because the head of the distribution is exactly where §5 and §7 show the pure form strains hardest.

Which fifty books. Every text here is a canonical, frequently anthologised work. A corpus of fifty books drawn instead from Gutenberg's least-downloaded, most obscure titles, amateur translations, regional almanacs, forgotten technical manuals, would very likely show a wider spread of exponents and a lower survival rate under the goodness-of-fit test, for the unglamorous reason that professionally edited canonical prose is unusually consistent in register from first page to last. We did not test this, because building and justifying a "representative" sample of Gutenberg is a separate and much larger project than the one we set out to do.

None of this makes Zipf's law false. Every one of fifty books in fourteen languages across twenty-three centuries produced a rank exponent within a factor of 1.4 of every other book's, which, for a regularity nobody has ever cleanly derived from first principles, is genuinely remarkable and was true again with case preserved, with hyphens split differently, and inside four different rank windows. What is not true is that the exponent is one number, that the pure form is always adequate, or that two studies quoting "the Zipf exponent" to three decimal places are necessarily describing the same measurement. We came away believing the regularity is real and the certainty usually attached to it in casual summary is not.

Reproducing this

Everything in this article comes from one file and one command. You need Python 3.12 and numpy; no other package is imported.

python zipf-in-books.py > zipf-in-books-output.txt

Expect roughly 4 to 5 minutes on a laptop once the fifty books are cached under analysis/data/gutenberg/ (the first run additionally downloads about 20 MB from gutenberg.org). Our archived run took 265.7 seconds on cached data; a fresh rerun of the identical file for this publication, on the same machine, took 611.6 seconds, all of the extra time spent re-tokenising rather than re-fitting, and produced byte-identical numbers apart from the two printed timing lines. The master seed is 20260614, hard-coded at the top of the file, and every random stream is spawned from it through numpy's SeedSequence, so the fitted values, the validation results and the figure data are deterministic. A rerun that disagrees with the archived output by more than sampling noise means something changed and we would like to hear about it. The interactive model lets you plot any of six of the cached books' real rank-frequency curves against both fitted forms in your own browser, and compare all fifty books' exponents by language and century.

References