============================================================================== COUNTING EVERY WORD IN FIFTY BOOKS TO TEST A LAW NOBODY CAN EXPLAIN Science Journaling Club, Volume 2 Issue 4, Summer 2026 ============================================================================== run date : 2026-09-14 python : 3.12.3 numpy : 2.4.2 master seed : 20260614 data : Project Gutenberg, https://www.gutenberg.org/ cache : analysis/data/gutenberg/ The club has no laboratory. This program is the experiment: it counts words in fifty public-domain books and fits two families of distribution to the counts by maximum likelihood. Nothing here was observed anywhere but in those fifty files. ============================================================================== VALIDATION A. The Hurwitz zeta function against closed forms ============================================================================== The whole likelihood rests on zeta(s, q). It is computed here by Euler-Maclaurin with 16 direct terms and four Bernoulli corrections. Club value beside the analytic value, for cases where one exists. case club accepted difference zeta(2,1) = pi^2/6 1.644934066848224 1.644934066848226 -2.220e-15 zeta(4,1) = pi^4/90 1.082323233711138 1.082323233711138 -2.220e-16 zeta(6,1) = pi^6/945 1.017343061984449 1.017343061984449 -2.220e-16 zeta(3,1) = Apery 1.202056903159593 1.202056903159594 -8.882e-16 zeta(1.5,1) 2.612375348685485 2.612375348685488 -3.109e-15 zeta(2,5) = pi^2/6 - sum_{k=1..4} k^-2 0.221322955737115 0.221322955737115 -1.665e-16 zeta(2.5,7.5) vs 4e6-term sum 0.035882887418555 0.035882887418556 -2.082e-17 Worst relative error over all cases: 1.35e-15 VERDICT: PASS, agreement to better than 1e-12 ============================================================================== VALIDATION B. The discrete power-law sampler against its own CDF ============================================================================== b = 1.95, xmin = 7, n = 2,000,000 draws KS distance from the exact model CDF : 0.000646 95% KS critical value 1.358/sqrt(n) : 0.000960 VERDICT: PASS, the sampler is drawing from the law it claims Analytic mean at b = 2.5, xmin = 7 is zeta(1.5,7)/zeta(2.5,7). club mean 19.47901 +/- 0.12387, analytic 19.55722, difference -0.07821 = -0.63 SE VERDICT: PASS ============================================================================== VALIDATION C. Recovering a known exponent from synthetic data ============================================================================== Draws from a discrete power law with a known b, fitted by the same MLE used on the books. xmin is held at its true value here so that the estimator is tested on its own, without the xmin search on top of it. reps = 300 per cell. z is the bias divided by its own standard error; |z| > 3 would mean a real bias rather than Monte Carlo noise. b_true n mean b bias SD(b) mean SE z 1.60 250 1.6039 +0.0039 0.0350 0.0382 +1.95 1.60 1000 1.6022 +0.0022 0.0188 0.0190 +2.01 1.60 4000 1.6005 +0.0005 0.0099 0.0095 +0.82 1.60 16000 1.6001 +0.0001 0.0045 0.0047 +0.46 1.60 64000 1.5997 -0.0003 0.0024 0.0024 -2.12 1.95 250 1.9508 +0.0008 0.0619 0.0602 +0.21 1.95 1000 1.9477 -0.0023 0.0293 0.0300 -1.37 1.95 4000 1.9488 -0.0012 0.0140 0.0150 -1.52 1.95 16000 1.9501 +0.0001 0.0071 0.0075 +0.16 1.95 64000 1.9501 +0.0001 0.0039 0.0038 +0.46 2.30 250 2.3019 +0.0019 0.0866 0.0825 +0.39 2.30 1000 2.3041 +0.0041 0.0418 0.0413 +1.71 2.30 4000 2.2990 -0.0010 0.0213 0.0206 -0.82 2.30 16000 2.2999 -0.0001 0.0104 0.0103 -0.16 2.30 64000 2.3000 +0.0000 0.0053 0.0051 +0.05 3.00 250 3.0181 +0.0181 0.1377 0.1281 +2.28 3.00 1000 2.9994 -0.0006 0.0617 0.0635 -0.16 3.00 4000 2.9979 -0.0021 0.0315 0.0317 -1.15 3.00 16000 3.0006 +0.0006 0.0162 0.0159 +0.65 3.00 64000 3.0003 +0.0003 0.0084 0.0079 +0.58 Largest |z| over the twenty cells: 2.28 The asymptotic standard error tracks the observed scatter, which is what licenses quoting SEs for the books without bootstrapping each one. FIGDATA convergence n mean sd se FIGDATA conv 250 1.95076 0.06187 0.06018 FIGDATA conv 1000 1.94768 0.02929 0.02999 FIGDATA conv 4000 1.94877 0.01400 0.01501 FIGDATA conv 16000 1.95006 0.00713 0.00752 FIGDATA conv 64000 1.95010 0.00386 0.00376 ============================================================================== VALIDATION D. Is the goodness-of-fit test calibrated? ============================================================================== A p-value is only worth printing if it is uniform when the null is true. Here: 80 synthetic vocabularies of 6,000 word types whose counts really are a discrete power law (b = 1.95, xmin = 1), each put through the whole procedure, xmin search included, with 80 bootstrap replicates. rejection rate at p < 0.05 : 0.062 (nominal 0.050, SE 0.024, z = +0.51) rejection rate at p <= 0.10: 0.150 (nominal 0.100) mean p = 0.505 (uniform would give 0.500), median p = 0.519 VERDICT: PASS, the test does not reject data that obey the law And the other half of calibration: does it reject when it should? 40 synthetic vocabularies of the same size drawn from a log-normal, which is the classic look-alike for a power law. power against log-normal counts at p < 0.05: 25 of 40 = 0.62 ============================================================================== VALIDATION E. The rank-side fit, and the price of ranking your own data ============================================================================== A synthetic book: N tokens drawn from a Zipf-Mandelbrot distribution over V types with known a and q. The fit then does what it does to a real book, which is to rank the types by their observed counts. Those ranks are estimates, and the noise in them pushes the exponent up. This is the measurement of that bias, not an argument that it is small. truth a = 1.00, q = 0.0, V = 20,000, N = 200,000 tokens, 20 replicates Zipf-Mandelbrot a : 0.9921 +/- 0.0010 bias -0.0079 (-0.8%) Zipf-Mandelbrot q : -0.028 +/- 0.009 truth 0.000 pure Zipf a : 0.9942 +/- 0.0006 bias -0.0058 (-0.6%) truth a = 1.05, q = 2.0, V = 20,000, N = 200,000 tokens, 20 replicates Zipf-Mandelbrot a : 1.0404 +/- 0.0010 bias -0.0096 (-0.9%) Zipf-Mandelbrot q : 1.924 +/- 0.035 truth 2.000 pure Zipf a : 0.9620 +/- 0.0008 bias -0.0880 (-8.4%) truth a = 1.00, q = 0.0, V = 20,000, N = 20,000 tokens, 20 replicates Zipf-Mandelbrot a : 0.9055 +/- 0.0033 bias -0.0945 (-9.5%) Zipf-Mandelbrot q : -0.332 +/- 0.022 truth 0.000 pure Zipf a : 0.9345 +/- 0.0026 bias -0.0655 (-6.6%) Read the third block against the first: with ten times fewer tokens the bias grows, because rank noise grows. Every exponent quoted for a real book below carries a bias of this kind and this size. It is the same sign and roughly the same size for every book, so comparisons between books survive it far better than any single value does. ============================================================================== THE CORPUS. Fifty works from Project Gutenberg ============================================================================== Source https://www.gutenberg.org/ , one plain-text file per work at https://www.gutenberg.org/cache/epub//pg.txt Title, author, language and release date are read from the file header. Bytes are counted before Gutenberg boilerplate is stripped. RETRIEVED 2026-09-14 MANIFEST written to analysis/data/gutenberg/MANIFEST.csv # PG ID lang title (from the file header) author PG release kB sha256[:8] 1 1122 en Hamlet William Shakespeare December 1, 1 186.8 aac0eb6c 2 20 en Paradise Lost John Milton October 1, 19 482.0 67321f47 3 521 en The Life and Adventures of Robinso Daniel Defoe May 1, 1996 638.7 6c5c1dc7 4 829 en Gulliver's Travels into Several Re Jonathan Swift February 1, 1 599.6 fca36de3 5 1342 en Pride and Prejudice Jane Austen June 1, 1998 754.3 3f6bb9d6 6 158 en Emma Jane Austen August 1, 199 911.8 532b122b 7 84 en Frankenstein; or, the modern prome Mary Wollstonecraft Shelle October 1, 19 438.4 7810cd48 8 2701 en Moby Dick; Or, The Whale Herman Melville July 1, 2001 1246.4 907420db 9 98 en A Tale of Two Cities Charles Dickens January 1, 19 788.3 d54c2b80 10 1228 en On the Origin of Species By Means Charles Darwin March 1, 1998 947.9 ededa9c0 11 1400 en Great Expectations Charles Dickens July 1, 1998 1033.4 9a637118 12 11 en Alice's Adventures in Wonderland Lewis Carroll June 27, 2008 170.2 01b38ea4 13 74 en The Adventures of Tom Sawyer, Comp Mark Twain July 1, 2004 424.2 74d77384 14 76 en Adventures of Huckleberry Finn Mark Twain June 29, 2004 607.9 d617a37a 15 1661 en The Adventures of Sherlock Holmes Arthur Conan Doyle March 1, 1999 593.4 922e2a12 16 345 en Dracula Bram Stoker October 1, 19 869.5 96cd16ea 17 219 en Heart of Darkness Joseph Conrad January 9, 20 231.5 c0b0bc91 18 4300 en Ulysses James Joyce July 1, 2003 1549.2 e0309462 19 64317 en The Great Gatsby F. Scott Fitzgerald January 17, 2 299.4 ce760ec3 20 6318 fr L'Avare Molière August 1, 200 162.7 b8a9edae 21 4650 fr Candide, ou l'optimisme Voltaire November 1, 2 228.6 1cc10ab3 22 13951 fr Les trois mousquetaires Alexandre Dumas November 4, 2 1416.6 46c01913 23 14155 fr Madame Bovary Gustave Flaubert November 26, 734.4 93f0a225 24 6099 fr Les Fleurs du Mal Charles Baudelaire July 1, 2004 179.4 67f9ede9 25 17489 fr Les misérables Tome I: Fantine Victor Hugo January 10, 2 693.8 a5de514b 26 5097 fr Vingt mille Lieues Sous Les Mers — Jules Verne February 1, 2 920.5 53507b02 27 2650 fr Du côté de chez Swann Marcel Proust May 1, 2001 1059.0 9ebdb837 28 2407 de Die Leiden des jungen Werther — Ba Johann Wolfgang von Goethe November 1, 2 129.7 7fe92c00 29 2229 de Faust: Der Tragödie erster Teil Johann Wolfgang von Goethe June 1, 2000 213.3 fed2a58e 30 7205 de Also sprach Zarathustra: Ein Buch Friedrich Wilhelm Nietzsch January 1, 20 552.0 4f71adfe 31 5323 de Effi Briest Theodor Fontane March 1, 2004 618.9 15a93db4 32 22367 de Die Verwandlung Franz Kafka August 21, 20 142.6 359d3f59 33 320 es Vida De Lazarillo De Tormes Y De S Anonymous September 1, 128.1 0abbdbd8 34 2000 es Don Quijote Miguel de Cervantes Saaved December 1, 1 2173.6 534f41d5 35 17340 es Marianela Benito Pérez Galdós December 17, 319.8 ec6e5dbd 36 1000 it La Divina Commedia di Dante: Compl Dante Alighieri August 1, 199 583.9 4669dcc0 37 52484 it Le avventure di Pinocchio: Storia Carlo Collodi July 3, 2016 282.1 b51a398d 38 54829 pt Memorias Posthumas de Braz Cubas Machado de Assis June 2, 2017 393.8 edfe4370 39 55752 pt Dom Casmurro Machado de Assis October 15, 2 408.4 0fc3dbf3 40 15975 nl Camera Obscura Hildebrand June 3, 2005 986.5 10ab7142 41 11024 nl Max Havelaar Multatuli February 1, 2 777.8 9cf9f2ef 42 218 la C. Iuli Caesaris De Bello Gallico, Julius Caesar July 3, 2008 165.9 84ac8411 43 227 la Aeneidos Virgil March 1, 1995 484.3 2620ba82 44 7000 fi Kalevala Elias Lönnrot November 1, 2 643.0 23e57fdc 45 11940 fi Seitsemän veljestä Aleksis Kivi April 1, 2004 675.3 b87d2858 46 30078 sv Hemsöborna August Strindberg September 25, 282.7 07aef01b 47 10686 da Tine Herman Bang January 1, 20 292.5 cddc3e56 48 43777 hu Az uj földesur (1. kötet) Mór Jókai September 20, 208.9 87607794 49 17839 el Οιδίπους Τύραννος Sophocles February 23, 181.8 d406e312 50 11511 eo Robinsono Kruso Daniel Defoe March 1, 2004 143.8 22b51004 Tokenised 50 books, 4,714,004 tokens in total, in 5.3 s. ============================================================================== DATA QUALITY. Is each file actually in the language it claims? ============================================================================== A crude but effective gate: the share of tokens that are common English function words. English books should score high, everything else low. A non-English file scoring high would mean English editorial matter has leaked into the text and is being counted as if it were the work. title lang EN share flag Hamlet en 0.1419 Paradise Lost en 0.1704 Robinson Crusoe en 0.1944 Gulliver's Travels en 0.1992 Pride and Prejudice en 0.1634 Emma en 0.1507 Frankenstein en 0.1986 Moby-Dick en 0.1850 A Tale of Two Cities en 0.1844 Origin of Species en 0.2093 Great Expectations en 0.1685 Alice in Wonderland en 0.1662 Tom Sawyer en 0.1691 Huckleberry Finn en 0.1592 Sherlock Holmes en 0.1745 Dracula en 0.1713 Heart of Darkness en 0.1731 Ulysses en 0.1605 The Great Gatsby en 0.1583 L'Avare fr 0.0005 Candide fr 0.0000 Les trois mousquetaires fr 0.0000 Madame Bovary fr 0.0000 Les Fleurs du Mal fr 0.0001 Les miserables I fr 0.0000 Vingt mille lieues fr 0.0001 Du cote de chez Swann fr 0.0000 Werther I de 0.0002 Faust I de 0.0000 Also sprach Zarathustra de 0.0004 Effi Briest de 0.0000 Die Verwandlung de 0.0001 Lazarillo de Tormes es 0.0000 Don Quijote es 0.0000 Marianela es 0.0000 Divina Commedia it 0.0000 Pinocchio it 0.0000 Bras Cubas pt 0.0000 Dom Casmurro pt 0.0000 Camera Obscura nl 0.0045 Max Havelaar nl 0.0046 De Bello Gallico I-IV la 0.0000 Aeneis la 0.0000 Kalevala fi 0.0001 Seitseman veljesta fi 0.0000 Hemsoborna sv 0.0004 Tine da 0.0015 Az uj foldesur I hu 0.0000 Oidipous Tyrannos (mod. Greek tran el 0.0001 Robinsono Kruso (Esperanto trans.) eo 0.0018 Nothing flagged. ============================================================================== PER-BOOK FITS ============================================================================== Rank side: pure Zipf p(r) ~ r^-a and Zipf-Mandelbrot p(r) ~ (r+q)^-a, both by exact multinomial maximum likelihood over all V ranks. dAIC is AIC(Zipf) - AIC(Zipf-Mandelbrot); positive means the shifted form wins. Frequency side: Clauset MLE for b with xmin by minimum KS, and a goodness-of-fit p from 200 parametric bootstrap replicates each. p <= 0.10 is the Clauset rule for ruling the pure power law out. title tokens types hapax a_zipf SE a_ZM q dAIC b xmin p_GOF Hamlet 30,314 4,938 0.607 0.9397 0.0022 1.1406 5.79 3314 1.963 1 0.005 Paradise Lost 80,169 10,096 0.508 0.9471 0.0013 1.0505 2.65 3940 1.953 5 0.475 Robinson Crusoe 121,569 6,069 0.377 1.0242 0.0011 1.2418 4.73 15616 1.825 6 0.925 Gulliver's Travels 105,375 8,186 0.422 0.9995 0.0011 1.1181 2.37 6223 1.976 10 0.980 Pride and Prejudice 127,858 6,822 0.387 0.9944 0.0010 1.2508 7.40 20861 1.907 20 0.140 Emma 161,111 7,201 0.388 1.0041 0.0009 1.3078 9.42 33389 1.792 8 0.010 Frankenstein 75,201 7,013 0.430 0.9886 0.0013 1.0849 1.77 3214 1.963 5 0.005 Moby-Dick 216,603 17,371 0.441 0.9983 0.0007 1.0817 1.81 8037 1.958 8 0.170 A Tale of Two Cities 137,429 9,849 0.435 1.0060 0.0010 1.1499 3.26 10538 1.863 4 0.690 Origin of Species 155,796 6,947 0.326 0.9992 0.0009 1.1210 2.37 7809 1.872 13 0.000 Great Expectations 186,711 10,966 0.425 1.0229 0.0008 1.2201 4.98 25035 1.821 4 0.205 Alice in Wonderland 26,776 2,636 0.431 0.9519 0.0025 1.1435 4.04 1875 1.921 7 0.030 Tom Sawyer 72,187 7,471 0.484 0.9798 0.0014 1.1076 2.82 4367 1.853 2 0.000 Huckleberry Finn 113,358 6,352 0.426 1.0122 0.0011 1.2191 4.77 12327 1.853 13 0.115 Sherlock Holmes 105,286 7,946 0.444 1.0023 0.0011 1.1781 4.16 10697 1.923 9 0.685 Dracula 162,259 9,407 0.453 1.0176 0.0009 1.2351 5.72 22193 1.874 17 0.125 Heart of Darkness 38,774 5,455 0.525 0.9684 0.0019 1.0674 1.86 1504 1.976 3 0.125 Ulysses 264,493 29,738 0.545 0.9873 0.0006 1.0491 1.44 7673 1.953 5 0.225 The Great Gatsby 48,676 5,938 0.531 0.9640 0.0017 1.1008 3.10 3155 1.824 1 0.020 L'Avare 22,150 3,376 0.581 0.9474 0.0027 1.1687 5.55 2443 1.913 1 0.115 Candide 34,450 5,829 0.576 0.9369 0.0020 1.0721 3.41 2212 1.970 2 0.960 Les trois mousquetaires 223,883 15,329 0.473 1.0006 0.0007 1.1978 6.33 31777 1.828 5 0.020 Madame Bovary 112,892 14,140 0.527 0.9718 0.0010 1.0845 2.96 7735 1.963 4 0.635 Les Fleurs du Mal 24,855 6,166 0.635 0.8975 0.0024 0.9591 1.36 505 2.095 1 0.135 Les miserables I 111,887 12,899 0.537 0.9752 0.0010 1.1139 3.89 9690 1.894 2 0.430 Vingt mille lieues 140,766 15,750 0.511 0.9735 0.0009 1.0831 2.92 8930 1.998 9 0.745 Du cote de chez Swann 168,591 16,613 0.531 0.9943 0.0008 1.1530 4.75 18109 1.898 4 0.840 Werther I 17,734 4,042 0.645 0.9053 0.0030 1.0221 2.77 805 2.078 1 0.000 Faust I 30,693 6,308 0.631 0.9062 0.0022 1.0523 4.56 2341 1.980 2 0.195 Also sprach Zarathustra 84,322 9,450 0.537 0.9575 0.0012 1.1433 6.17 8437 1.827 1 0.135 Effi Briest 95,381 11,192 0.598 0.9759 0.0011 1.1901 7.44 13398 1.864 2 0.015 Die Verwandlung 19,230 3,790 0.626 0.9107 0.0029 1.0789 4.52 1500 1.931 2 0.095 Lazarillo de Tormes 20,096 3,724 0.611 0.9562 0.0028 1.0424 1.44 580 1.984 1 0.105 Don Quijote 383,633 22,943 0.486 1.0446 0.0005 1.1513 2.19 26293 1.771 2 0.000 Marianela 50,991 8,224 0.595 0.9699 0.0016 1.0572 1.75 2005 1.969 1 0.970 Divina Commedia 97,718 13,636 0.584 0.9593 0.0011 1.0596 2.63 5068 1.923 1 0.225 Pinocchio 41,536 6,220 0.531 0.9340 0.0019 1.0411 2.51 1909 1.928 2 0.030 Bras Cubas 63,347 10,138 0.563 0.9632 0.0014 1.0391 1.58 2239 2.012 2 0.225 Dom Casmurro 67,239 8,855 0.537 0.9709 0.0014 1.0811 2.47 3961 1.928 2 0.105 Camera Obscura 161,385 18,875 0.568 0.9903 0.0008 1.1099 3.22 12942 1.902 1 0.120 Max Havelaar 125,009 13,814 0.537 0.9794 0.0010 1.1276 4.30 12512 1.934 5 0.860 De Bello Gallico I-IV 20,542 5,688 0.608 0.8102 0.0028 0.8814 2.38 434 2.081 2 0.205 Aeneis 63,791 16,628 0.556 0.7763 0.0015 0.7835 0.23 18 2.485 18 0.660 Kalevala 67,782 20,159 0.609 0.7571 0.0015 0.8311 6.11 1715 2.109 2 0.000 Seitseman veljesta 81,840 21,794 0.643 0.8591 0.0012 0.8560 -0.07 5 2.112 1 0.695 Hemsoborna 43,433 8,492 0.652 0.9511 0.0017 1.0387 1.94 1540 2.089 1 0.000 Tine 47,640 5,953 0.561 0.9632 0.0017 1.1709 5.93 5228 1.866 1 0.860 Az uj foldesur I 26,002 8,604 0.748 0.8768 0.0023 0.8560 -0.31 85 2.020 4 0.930 Oidipous Tyrannos (mod. Greek tr 13,889 3,663 0.676 0.8781 0.0035 1.0093 3.53 793 2.147 1 0.000 Robinsono Kruso (Esperanto trans 21,352 4,157 0.607 0.9470 0.0027 1.0067 0.96 331 1.997 1 0.750 ============================================================================== RESULTS TABLE. Fifty books, one row each ============================================================================== title lang year tokens types a_zipf SE a_ZM q b xmin p_GOF hapax Hamlet en 1601 30,314 4,938 0.9397 0.0022 1.1406 5.79 1.963 1 0.005 0.607 Paradise Lost en 1667 80,169 10,096 0.9471 0.0013 1.0505 2.65 1.953 5 0.475 0.508 Robinson Crusoe en 1719 121,569 6,069 1.0242 0.0011 1.2418 4.73 1.825 6 0.925 0.377 Gulliver's Travels en 1726 105,375 8,186 0.9995 0.0011 1.1181 2.37 1.976 10 0.980 0.422 Pride and Prejudice en 1813 127,858 6,822 0.9944 0.0010 1.2508 7.40 1.907 20 0.140 0.387 Emma en 1815 161,111 7,201 1.0041 0.0009 1.3078 9.42 1.792 8 0.010 0.388 Frankenstein en 1818 75,201 7,013 0.9886 0.0013 1.0849 1.77 1.963 5 0.005 0.430 Moby-Dick en 1851 216,603 17,371 0.9983 0.0007 1.0817 1.81 1.958 8 0.170 0.441 A Tale of Two Cities en 1859 137,429 9,849 1.0060 0.0010 1.1499 3.26 1.863 4 0.690 0.435 Origin of Species en 1859 155,796 6,947 0.9992 0.0009 1.1210 2.37 1.872 13 0.000 0.326 Great Expectations en 1861 186,711 10,966 1.0229 0.0008 1.2201 4.98 1.821 4 0.205 0.425 Alice in Wonderland en 1865 26,776 2,636 0.9519 0.0025 1.1435 4.04 1.921 7 0.030 0.431 Tom Sawyer en 1876 72,187 7,471 0.9798 0.0014 1.1076 2.82 1.853 2 0.000 0.484 Huckleberry Finn en 1884 113,358 6,352 1.0122 0.0011 1.2191 4.77 1.853 13 0.115 0.426 Sherlock Holmes en 1892 105,286 7,946 1.0023 0.0011 1.1781 4.16 1.923 9 0.685 0.444 Dracula en 1897 162,259 9,407 1.0176 0.0009 1.2351 5.72 1.874 17 0.125 0.453 Heart of Darkness en 1899 38,774 5,455 0.9684 0.0019 1.0674 1.86 1.976 3 0.125 0.525 Ulysses en 1922 264,493 29,738 0.9873 0.0006 1.0491 1.44 1.953 5 0.225 0.545 The Great Gatsby en 1925 48,676 5,938 0.9640 0.0017 1.1008 3.10 1.824 1 0.020 0.531 L'Avare fr 1668 22,150 3,376 0.9474 0.0027 1.1687 5.55 1.913 1 0.115 0.581 Candide fr 1759 34,450 5,829 0.9369 0.0020 1.0721 3.41 1.970 2 0.960 0.576 Les trois mousquetaires fr 1844 223,883 15,329 1.0006 0.0007 1.1978 6.33 1.828 5 0.020 0.473 Madame Bovary fr 1857 112,892 14,140 0.9718 0.0010 1.0845 2.96 1.963 4 0.635 0.527 Les Fleurs du Mal fr 1857 24,855 6,166 0.8975 0.0024 0.9591 1.36 2.095 1 0.135 0.635 Les miserables I fr 1862 111,887 12,899 0.9752 0.0010 1.1139 3.89 1.894 2 0.430 0.537 Vingt mille lieues fr 1870 140,766 15,750 0.9735 0.0009 1.0831 2.92 1.998 9 0.745 0.511 Du cote de chez Swann fr 1913 168,591 16,613 0.9943 0.0008 1.1530 4.75 1.898 4 0.840 0.531 Werther I de 1774 17,734 4,042 0.9053 0.0030 1.0221 2.77 2.078 1 0.000 0.645 Faust I de 1808 30,693 6,308 0.9062 0.0022 1.0523 4.56 1.980 2 0.195 0.631 Also sprach Zarathustra de 1883 84,322 9,450 0.9575 0.0012 1.1433 6.17 1.827 1 0.135 0.537 Effi Briest de 1895 95,381 11,192 0.9759 0.0011 1.1901 7.44 1.864 2 0.015 0.598 Die Verwandlung de 1915 19,230 3,790 0.9107 0.0029 1.0789 4.52 1.931 2 0.095 0.626 Lazarillo de Tormes es 1554 20,096 3,724 0.9562 0.0028 1.0424 1.44 1.984 1 0.105 0.611 Don Quijote es 1605 383,633 22,943 1.0446 0.0005 1.1513 2.19 1.771 2 0.000 0.486 Marianela es 1878 50,991 8,224 0.9699 0.0016 1.0572 1.75 1.969 1 0.970 0.595 Divina Commedia it 1320 97,718 13,636 0.9593 0.0011 1.0596 2.63 1.923 1 0.225 0.584 Pinocchio it 1883 41,536 6,220 0.9340 0.0019 1.0411 2.51 1.928 2 0.030 0.531 Bras Cubas pt 1881 63,347 10,138 0.9632 0.0014 1.0391 1.58 2.012 2 0.225 0.563 Dom Casmurro pt 1899 67,239 8,855 0.9709 0.0014 1.0811 2.47 1.928 2 0.105 0.537 Camera Obscura nl 1839 161,385 18,875 0.9903 0.0008 1.1099 3.22 1.902 1 0.120 0.568 Max Havelaar nl 1860 125,009 13,814 0.9794 0.0010 1.1276 4.30 1.934 5 0.860 0.537 De Bello Gallico I-IV la -50 20,542 5,688 0.8102 0.0028 0.8814 2.38 2.081 2 0.205 0.608 Aeneis la -19 63,791 16,628 0.7763 0.0015 0.7835 0.23 2.485 18 0.660 0.556 Kalevala fi 1849 67,782 20,159 0.7571 0.0015 0.8311 6.11 2.109 2 0.000 0.609 Seitseman veljesta fi 1870 81,840 21,794 0.8591 0.0012 0.8560 -0.07 2.112 1 0.695 0.643 Hemsoborna sv 1887 43,433 8,492 0.9511 0.0017 1.0387 1.94 2.089 1 0.000 0.652 Tine da 1889 47,640 5,953 0.9632 0.0017 1.1709 5.93 1.866 1 0.860 0.561 Az uj foldesur I hu 1863 26,002 8,604 0.8768 0.0023 0.8560 -0.31 2.020 4 0.930 0.748 Oidipous Tyrannos (mod. Greek tr el -429 13,889 3,663 0.8781 0.0035 1.0093 3.53 2.147 1 0.000 0.676 Robinsono Kruso (Esperanto trans eo 1719 21,352 4,157 0.9470 0.0027 1.0067 0.96 1.997 1 0.750 0.607 ============================================================================== DOES THE EXPONENT MEAN ANYTHING? Heterogeneity across the fifty books ============================================================================== Unweighted mean a over 50 books : 0.9549, SD 0.0597, range 0.7571 to 1.0446 Inverse-variance weighted mean : 0.9834 Cochran Q : 91381.6 on 49 degrees of freedom Q/df : 1864.9 (1.0 would mean the books agree) p : 0 The formal standard errors are tiny because the likelihood pretends every token is an independent draw. They are not a believable yardstick for comparing books, and section 9 replaces them with a measured one. By language (original-language texts only where marked): language n mean a SD min max Danish 1 0.9632 n/a 0.9632 0.9632 German 5 0.9311 0.0332 0.9053 0.9759 Greek 1 0.8781 n/a 0.8781 0.8781 English 19 0.9899 0.0252 0.9397 1.0242 Esperanto 1 0.9470 n/a 0.9470 0.9470 Spanish 3 0.9902 0.0476 0.9562 1.0446 Finnish 2 0.8081 0.0721 0.7571 0.8591 French 8 0.9622 0.0337 0.8975 1.0006 Hungarian 1 0.8768 n/a 0.8768 0.8768 Italian 2 0.9466 0.0179 0.9340 0.9593 Latin 2 0.7932 0.0239 0.7763 0.8102 Dutch 2 0.9849 0.0077 0.9794 0.9903 Portuguese 2 0.9671 0.0055 0.9632 0.9709 Swedish 1 0.9511 n/a 0.9511 0.9511 Same author, two books, same language: the tightest comparison available. Pride and Prejudice vs Emma da = -0.0097 z(formal) = -7.0 A Tale of Two Cities vs Great Expectations da = -0.0169 z(formal) = -13.4 Tom Sawyer vs Huckleberry Finn da = -0.0324 z(formal) = -18.4 Bras Cubas vs Dom Casmurro da = -0.0078 z(formal) = -3.9 Morphology. Finnish and Hungarian inflect heavily, so a given lemma appears as many distinct word forms and the head of the distribution is flattened. English and Chinese-style isolating languages do the opposite. English n=19 mean a = 0.9899 Finnish + Hungarian n=3 mean a = 0.8310 difference +0.1589 Welch t = +4.21 using between-book scatter as the error Type-token ratio, the same story without any fitting at all: English mean types per token 0.0858 Finnish mean types per token 0.2819 Hungarian mean types per token 0.3309 Latin mean types per token 0.2688 Esperanto mean types per token 0.1947 Century trend, 45 original-language works from 1554 to 1925: d a / d century = -0.0023 +/- 0.0086 (t = -0.27, p = 0.791) ============================================================================== FAILURE MODE 1. The exponent is not one number, it drifts with rank ============================================================================== The same pure Zipf law fitted inside four rank windows, conditional on the rank falling in the window. If a single power law described the whole distribution these four numbers would agree. title r 1-100 r 101-1000 r 1001-10000 r >10000 Hamlet 0.691 1.224 0.764 n/a Paradise Lost 0.776 1.017 1.110 0.200 Robinson Crusoe 0.799 1.218 1.508 n/a Gulliver's Travels 0.847 1.016 1.328 n/a Pride and Prejudice 0.720 1.101 1.507 n/a Emma 0.696 1.190 1.554 n/a Frankenstein 0.876 0.939 1.288 n/a Moby-Dick 0.844 1.144 1.132 0.200 A Tale of Two Cities 0.818 1.158 1.269 n/a Origin of Species 0.850 1.026 1.641 n/a Great Expectations 0.786 1.259 1.322 0.200 Alice in Wonderland 0.777 1.218 1.257 n/a Tom Sawyer 0.806 1.188 1.174 n/a Huckleberry Finn 0.787 1.246 1.461 n/a Sherlock Holmes 0.780 1.088 1.314 n/a Dracula 0.766 1.207 1.379 n/a Heart of Darkness 0.843 1.090 1.014 n/a Ulysses 0.858 1.014 1.091 0.761 The Great Gatsby 0.800 1.108 1.079 n/a L'Avare 0.708 1.217 0.679 n/a Candide 0.741 1.051 0.879 n/a Les trois mousquetaires 0.724 1.106 1.316 0.200 Madame Bovary 0.786 1.005 1.108 0.200 Les Fleurs du Mal 0.777 0.888 0.712 n/a Les miserables I 0.754 1.041 1.158 0.200 Vingt mille lieues 0.776 1.017 1.155 0.200 Du cote de chez Swann 0.731 1.072 1.172 0.200 Werther I 0.732 1.107 0.538 n/a Faust I 0.674 0.999 0.771 n/a Also sprach Zarathustra 0.695 1.131 1.131 n/a Effi Briest 0.670 1.153 1.064 0.200 Die Verwandlung 0.702 1.123 0.575 n/a Lazarillo de Tormes 0.863 1.038 0.615 n/a Don Quijote 0.894 1.045 1.275 0.831 Marianela 0.850 1.052 0.888 n/a Divina Commedia 0.781 1.089 1.133 0.200 Pinocchio 0.792 1.070 1.004 n/a Bras Cubas 0.859 0.994 0.943 0.200 Dom Casmurro 0.817 1.039 1.033 n/a Camera Obscura 0.779 1.039 1.139 0.200 Max Havelaar 0.742 1.030 1.179 0.200 De Bello Gallico I-IV 0.684 0.919 0.758 n/a Aeneis 0.731 0.712 1.003 0.200 Kalevala 0.581 0.744 0.971 0.200 Seitseman veljesta 0.826 0.877 0.947 0.200 Hemsoborna 0.792 1.075 0.740 n/a Tine 0.708 1.158 1.000 n/a Az uj foldesur I 0.861 0.992 0.459 n/a Oidipous Tyrannos (mod. Greek tr 0.694 1.045 0.345 n/a Robinsono Kruso (Esperanto trans 0.868 1.031 0.633 n/a r 1-100 n = 50 mean 0.7743 SD 0.0669 r 101-1000 n = 50 mean 1.0662 SD 0.1135 r 1001-10000 n = 50 mean 1.0503 SD 0.2955 r >10000 n = 18 mean 0.2662 SD 0.1931 Within-book change from the head window to the r 1001-10000 window: mean +0.2761, SD 0.2934, positive in 41 of 50 books paired t = +6.65 ============================================================================== FAILURE MODE 2. A pure power law against Mandelbrot's shifted form ============================================================================== Zipf-Mandelbrot adds one parameter, the shift q, and nests the pure law at q = 0. Twice the log-likelihood difference is a chi-square on 1 df. Zipf-Mandelbrot beats pure Zipf on AIC in 50 of 50 books. Likelihood-ratio statistic: median 4166, min 7.3, max 33391 A chi-square on 1 df needs 3.84 for p = 0.05. 50 of 50 books exceed it. Fitted shift q: median 3.03, range -0.31 to 9.42 What the shift does to the exponent: mean a under pure Zipf 0.9549 mean a under Zipf-Mandelbrot 1.0866 mean change +0.1316, SD 0.0652 So the headline exponent moves by about 14% depending on which of two defensible models you fit. Anyone quoting a Zipf exponent to three decimals without saying which form they fitted is quoting noise. ============================================================================== FAILURE MODE 3. Does the pure power law actually fit? The GOF test ============================================================================== Clauset, Shalizi and Newman rule a power law out when p <= 0.10. books with p > 0.10 (power law not ruled out): 34 of 50 books with p >= 0.05 : 35 of 50 median p = 0.138 Frequency-side exponent b over the fifty books: mean 1.951, SD 0.117 restricted to books the test does not reject: mean 1.960, SD 0.118, n = 34 Two roads to the same exponent. a = 1/(b-1) from the frequency side against the rank-side a, per book: mean 1/(b-1) 1.0645 mean rank a 0.9549 mean difference +0.1096, SD 0.0761, correlation 0.790 They are measuring the same thing through different windows: the rank fit uses every type, the frequency fit uses only the 49% of types above xmin. ============================================================================== FAILURE MODE 4. Words that appear exactly once ============================================================================== Under a fitted model the expected number of types seen exactly once is sum_r N p_r (1-p_r)^(N-1). Books really are full of one-off words, and this is where a rank-side model is asked to predict something it was not fitted on. title observed Zipf pred. ZM pred. ZM/obs Hamlet 2998 1211 1294 0.432 Paradise Lost 5129 2224 2452 0.478 Robinson Crusoe 2289 624 1111 0.485 Gulliver's Travels 3451 1391 1714 0.497 Pride and Prejudice 2637 643 1261 0.478 Emma 2796 527 1306 0.467 Frankenstein 3015 1344 1540 0.511 Moby-Dick 7654 3264 3773 0.493 A Tale of Two Cities 4289 1614 2097 0.489 Origin of Species 2263 479 792 0.350 Great Expectations 4656 1573 2352 0.505 Alice in Wonderland 1135 421 546 0.481 Tom Sawyer 3617 1527 1759 0.486 Huckleberry Finn 2706 730 1200 0.444 Sherlock Holmes 3526 1321 1753 0.497 Dracula 4257 1256 1981 0.465 Heart of Darkness 2865 1306 1385 0.483 Ulysses 16195 7118 7523 0.465 The Great Gatsby 3154 1294 1451 0.460 L'Avare 1963 794 868 0.442 Candide 3360 1460 1535 0.457 Les trois mousquetaires 7256 2443 3461 0.477 Madame Bovary 7445 3350 3627 0.487 Les Fleurs du Mal 3916 1712 1719 0.439 Les miserables I 6932 2916 3261 0.470 Vingt mille lieues 8052 3516 3905 0.485 Du cote de chez Swann 8815 3619 4183 0.475 Werther I 2609 1090 1104 0.423 Faust I 3979 1658 1706 0.429 Also sprach Zarathustra 5070 1955 2321 0.458 Effi Briest 6695 2543 2885 0.431 Die Verwandlung 2374 969 1008 0.425 Lazarillo de Tormes 2277 976 996 0.437 Don Quijote 11151 4146 5093 0.457 Marianela 4894 2136 2196 0.449 Divina Commedia 7968 3332 3539 0.444 Pinocchio 3304 1454 1562 0.473 Bras Cubas 5708 2622 2698 0.473 Dom Casmurro 4752 2105 2261 0.476 Camera Obscura 10725 4546 4909 0.458 Max Havelaar 7413 3083 3489 0.471 De Bello Gallico I-IV 3458 1534 1559 0.451 Aeneis 9250 4272 4289 0.464 Kalevala 12274 5436 5551 0.452 Seitseman veljesta 14009 6129 6126 0.437 Hemsoborna 5535 2306 2327 0.420 Tine 3338 1317 1503 0.450 Az uj foldesur I 6435 2505 2513 0.391 Oidipous Tyrannos (mod. Greek tr 2476 1013 1015 0.410 Robinsono Kruso (Esperanto trans 2525 1103 1118 0.443 Zipf-Mandelbrot predicted / observed hapax count: mean 0.458, SD 0.030 under-predicts in 50 of 50 books ============================================================================== HOW BIG IS A REAL DIFFERENCE? Splitting every book in half ============================================================================== A book is not a bag of independent draws, so the likelihood's standard error understates the true variability. Measure it instead: fit the first half and the second half of each book separately. The scatter between halves of the same book is the instrument's real repeatability. title a first a second diff formal SE(half) Hamlet 0.9100 0.9041 +0.0059 0.0047 Paradise Lost 0.9174 0.9303 -0.0128 0.0026 Robinson Crusoe 1.0086 1.0051 +0.0035 0.0022 Gulliver's Travels 0.9834 0.9785 +0.0049 0.0023 Pride and Prejudice 0.9752 0.9746 +0.0007 0.0021 Emma 0.9845 0.9853 -0.0008 0.0019 Frankenstein 0.9643 0.9738 -0.0094 0.0028 Moby-Dick 0.9821 0.9809 +0.0013 0.0015 A Tale of Two Cities 0.9868 0.9877 -0.0009 0.0020 Origin of Species 0.9848 0.9812 +0.0036 0.0019 Great Expectations 1.0038 1.0069 -0.0030 0.0017 Alice in Wonderland 0.9225 0.9281 -0.0055 0.0053 Tom Sawyer 0.9564 0.9569 -0.0005 0.0029 Huckleberry Finn 0.9877 0.9946 -0.0068 0.0023 Sherlock Holmes 0.9827 0.9812 +0.0015 0.0023 Dracula 0.9989 0.9977 +0.0012 0.0018 Heart of Darkness 0.9421 0.9434 -0.0012 0.0041 Ulysses 0.9694 0.9696 -0.0002 0.0013 The Great Gatsby 0.9379 0.9357 +0.0022 0.0036 L'Avare 0.9095 0.9193 -0.0098 0.0057 Candide 0.9075 0.9081 -0.0006 0.0043 Les trois mousquetaires 0.9844 0.9800 +0.0044 0.0015 Madame Bovary 0.9523 0.9488 +0.0035 0.0021 Les Fleurs du Mal 0.8633 0.8672 -0.0040 0.0052 Les miserables I 0.9501 0.9565 -0.0063 0.0022 Vingt mille lieues 0.9509 0.9552 -0.0043 0.0019 Du cote de chez Swann 0.9727 0.9775 -0.0048 0.0017 Werther I 0.8656 0.8700 -0.0044 0.0064 Faust I 0.8669 0.8801 -0.0132 0.0046 Also sprach Zarathustra 0.9392 0.9251 +0.0141 0.0026 Effi Briest 0.9474 0.9521 -0.0047 0.0024 Die Verwandlung 0.8687 0.8803 -0.0117 0.0062 Lazarillo de Tormes 0.9246 0.9257 -0.0011 0.0060 Don Quijote 1.0303 1.0295 +0.0008 0.0011 Marianela 0.9396 0.9500 -0.0105 0.0034 Divina Commedia 0.9338 0.9367 -0.0028 0.0023 Pinocchio 0.9067 0.9049 +0.0018 0.0039 Bras Cubas 0.9397 0.9391 +0.0006 0.0030 Dom Casmurro 0.9456 0.9485 -0.0029 0.0029 Camera Obscura 0.9701 0.9698 +0.0004 0.0017 Max Havelaar 0.9637 0.9524 +0.0112 0.0020 De Bello Gallico I-IV 0.7746 0.7671 +0.0075 0.0061 Aeneis 0.7369 0.7427 -0.0058 0.0032 Kalevala 0.7121 0.7226 -0.0105 0.0032 Seitseman veljesta 0.8327 0.8249 +0.0078 0.0026 Hemsoborna 0.9213 0.9242 -0.0029 0.0037 Tine 0.9333 0.9346 -0.0013 0.0036 Az uj foldesur I 0.8391 0.8481 -0.0089 0.0050 Oidipous Tyrannos (mod. Greek tr 0.8416 0.8383 +0.0033 0.0074 Robinsono Kruso (Esperanto trans 0.9114 0.9247 -0.0133 0.0057 observed SD of the half-to-half difference : 0.0062 formal SE of that same difference : 0.0033 ratio : 1.9 Take 0.0044 as the honest one-book standard error, which is the half-to-half scatter divided by sqrt(2). It is 3 times the likelihood's own figure. Redo the heterogeneity test with it: Q = 9157 on 49 df, Q/df = 187, p = 0 The differences between books are still far larger than the noise. Same-author pairs against the measured scatter: Pride and Prejudice vs Emma da = -0.0097 z(measured) = -1.57 A Tale of Two Cities vs Great Expectations da = -0.0169 z(measured) = -2.73 Tom Sawyer vs Huckleberry Finn da = -0.0324 z(measured) = -5.25 Bras Cubas vs Dom Casmurro da = -0.0078 z(measured) = -1.26 ============================================================================== CLUB VALUES BESIDE PUBLISHED VALUES ============================================================================== 1. Moby-Dick, the single most-fitted word-frequency data set in the power-law literature. Clauset, Shalizi and Newman (2009), Table 6.1, row 'count of word use', report for exactly this book: n(types) = 18,855 xmin = 7 +/- 2 b = 1.95 +/- 0.02 ntail = 2958 +/- 987 p = 0.49 The club, from its own download and its own tokeniser: n(types) = 17,371 xmin = 8 b = 1.958 +/- 0.018 ntail = 2754 p = 0.170 KS = 0.0127 difference in b: +0.008, which is 0.29 combined standard errors. AGREEMENT. The two studies disagree about xmin, 8 against 7, and that is the more interesting disagreement, so here is the same fit held at their xmin on our own token counts, which is the closest thing to a like-for-like comparison the two studies admit: club at xmin = 7: ntail = 3100, b = 1.9418 +/- 0.0169 Clauset et al. : ntail = 2958 +/- 987, b = 1.95 +/- 0.02 difference -0.0082, which is -0.31 combined standard errors. AGREEMENT. Our KS search lands on xmin = 8, one word past their 7; both are inside the +/- 2 they quote for their own bootstrap, so the two studies agree that this parameter is the soft one. The exponent is not: at a common xmin the two agree to 0.3 standard errors. Note on what can and cannot be compared here. Clauset et al. do not publish their tokeniser. Our vocabulary is 17,371 types against their 18,855, a difference of -7.9%, so the two studies are not counting quite the same objects and a small shift in b is expected on those grounds alone. The comparison is worth making anyway because it is the one number in this study that somebody else has published for the same book. 2. Moreno-Sanchez, Font-Clos and Corral (2016) fitted 31,075 English Gutenberg texts with the same MLE-plus-KS machinery and report a mean frequency-side exponent of about 2.03 with SD 0.15 among the texts their test did not reject, and roughly 40% of texts surviving the test. Club, English subset, n = 19: mean b = 1.898, SD 0.061 Club, all 50 books: mean b = 1.951, SD 0.117 difference from 2.03 on the English subset: -0.132 = -9.47 SE of our mean Club survival rate at p > 0.10: 12 of 19 English (63%), 34 of 50 overall (68%) 3. Zipf's own claim, and the textbook figure, is a rank exponent of about 1. Equivalently b = 1 + 1/a = 2. Club rank exponent, English books : mean 0.9899, SD 0.0252 Club rank exponent, all books : mean 0.9549, SD 0.0597 distance of the English mean from 1: -0.0101 in units of the measured one-book scatter (0.0044): 2.3 4. Ferrer-i-Cancho and Sole (2001) report two regimes, with the second regime steeper, near a = 2, beyond a crossover around rank 10^3 to 10^4. Club window r 1-100 : mean a = 0.774 Club window r 1001-10000 : mean a = 1.050 Same direction, same rough size. ============================================================================== SENSITIVITY. What a different tokeniser does to the answer ============================================================================== Tokeniser A, used everywhere above: letters with internal apostrophes, lowercased. B: letters only, so don't becomes don and t. C: letters with internal apostrophes and hyphens, so to-morrow stays one word. D: A without lowercasing, so The and the are different types. title A B - A C - A D - A Hamlet 0.9397 +0.0023 +0.0009 -0.0293 Paradise Lost 0.9471 +0.0057 +0.0006 -0.0298 Robinson Crusoe 1.0242 -0.0005 +0.0017 -0.0053 Gulliver's Travels 0.9995 -0.0001 +0.0014 -0.0082 Pride and Prejudice 0.9944 +0.0006 +0.0012 -0.0108 Emma 1.0041 +0.0006 +0.0017 -0.0107 Frankenstein 0.9886 -0.0003 +0.0008 -0.0103 Moby-Dick 0.9983 +0.0024 +0.0034 -0.0107 A Tale of Two Cities 1.0060 +0.0005 +0.0021 -0.0128 Origin of Species 0.9992 -0.0006 +0.0036 -0.0054 Great Expectations 1.0229 +0.0017 +0.0026 -0.0106 Alice in Wonderland 0.9519 +0.0051 +0.0018 -0.0136 Tom Sawyer 0.9798 +0.0078 +0.0022 -0.0150 Huckleberry Finn 1.0122 +0.0080 +0.0030 -0.0097 Sherlock Holmes 1.0023 -0.0000 +0.0026 -0.0137 Dracula 1.0176 +0.0002 +0.0016 -0.0121 Heart of Darkness 0.9684 +0.0008 +0.0022 -0.0149 Ulysses 0.9873 +0.0043 -0.0000 -0.0161 The Great Gatsby 0.9640 +0.0072 +0.0015 -0.0149 L'Avare 0.9474 +0.0129 -0.0034 -0.0189 Candide 0.9369 +0.0139 -0.0022 -0.0121 Les trois mousquetaires 1.0006 +0.0144 -0.0028 -0.0130 Madame Bovary 0.9718 +0.0167 -0.0027 -0.0124 Les Fleurs du Mal 0.8975 +0.0183 -0.0009 -0.0356 Les miserables I 0.9752 +0.0182 -0.0021 -0.0158 Vingt mille lieues 0.9735 +0.0127 -0.0024 -0.0149 Du cote de chez Swann 0.9943 +0.0188 -0.0014 -0.0072 Werther I 0.9053 +0.0016 +0.0000 -0.0162 Faust I 0.9062 +0.0025 +0.0000 -0.0428 Also sprach Zarathustra 0.9575 +0.0012 +0.0022 -0.0209 Effi Briest 0.9759 +0.0005 +0.0002 -0.0191 Die Verwandlung 0.9107 +0.0000 +0.0000 -0.0129 Lazarillo de Tormes 0.9562 +0.0000 +0.0000 -0.0112 Don Quijote 1.0446 +0.0000 +0.0000 -0.0065 Marianela 0.9699 +0.0000 +0.0001 -0.0158 Divina Commedia 0.9593 +0.0140 +0.0000 -0.0114 Pinocchio 0.9340 +0.0011 -0.0000 -0.0172 Bras Cubas 0.9632 +0.0006 -0.0068 -0.0142 Dom Casmurro 0.9709 +0.0001 -0.0051 -0.0156 Camera Obscura 0.9903 +0.0001 +0.0003 -0.0129 Max Havelaar 0.9794 +0.0003 -0.0006 -0.0139 De Bello Gallico I-IV 0.8102 -0.0000 +0.0000 -0.0094 Aeneis 0.7763 +0.0000 +0.0000 -0.0129 Kalevala 0.7571 +0.0011 -0.0001 -0.0170 Seitseman veljesta 0.8591 -0.0000 +0.0006 -0.0195 Hemsoborna 0.9511 -0.0000 +0.0001 -0.0100 Tine 0.9632 +0.0001 +0.0001 -0.0178 Az uj foldesur I 0.8768 +0.0000 -0.0000 -0.0208 Oidipous Tyrannos (mod. Greek tr 0.8781 +0.0000 +0.0000 -0.0179 Robinsono Kruso (Esperanto trans 0.9470 -0.0000 -0.0008 -0.0168 B letters only mean +0.0039 SD 0.0060 largest |change| 0.0188 C hyphens joined mean +0.0001 SD 0.0020 largest |change| 0.0068 D case kept mean -0.0151 SD 0.0070 largest |change| 0.0428 Set those against the 0.0044 half-to-half scatter: the choice of tokeniser moves the exponent by more than the measurement noise does, which is why comparing an exponent from one study with an exponent from another is only meaningful when both tokenised the same way. ============================================================================== FIGURE DATA. Every number the article's figures are drawn from ============================================================================== FIG1 rank-frequency curves, rank then count, log-spaced ranks FIG1 Moby-Dick|en|N=216603|V=17371 FIG1pts 1,14535 2,6624 3,6447 4,4739 6,4184 7,2990 10,1993 12,1778 16,1645 20,1320 26,1065 33,779 43,618 55,481 70,384 91,305 116,242 149,183 192,131 246,95 317,75 407,54 522,41 671,32 862,25 1107,19 1421,15 1826,12 2345,9 3012,7 3869,5 4969,4 6382,3 8198,2 10529,1 13524,1 17371,1 FIG1 Pride and Prejudice|en|N=127858|V=6822 FIG1pts 1,4663 2,4325 3,3844 4,3767 5,2284 6,2119 8,1994 10,1751 12,1590 15,1358 19,1186 24,879 30,699 37,536 47,455 59,352 74,262 92,206 116,153 145,115 182,89 229,72 287,60 360,44 451,34 566,27 709,21 890,15 1115,11 1399,8 1754,6 2200,4 2759,3 3459,2 4338,1 5440,1 6822,1 FIG1 Les miserables I|fr|N=111887|V=12899 FIG1pts 1,4472 2,3040 3,2949 4,2582 5,2548 7,1533 9,1350 11,1134 14,1036 18,843 23,721 30,579 38,466 49,303 62,230 79,164 101,125 128,102 163,80 208,61 266,50 339,38 431,29 550,23 701,17 894,13 1139,10 1452,8 1851,6 2359,4 3007,3 3833,3 4886,2 6228,1 7939,1 10119,1 12899,1 FIG1 Don Quijote|es|N=383633|V=22943 FIG1pts 1,20769 2,18410 3,18272 4,10492 5,9876 6,8285 8,6346 10,4752 13,3492 17,2714 22,2174 28,1808 37,1217 48,1015 62,685 80,487 103,386 133,308 172,244 223,186 288,143 373,108 482,82 624,61 807,47 1045,34 1351,25 1748,19 2261,14 2925,10 3784,7 4895,5 6333,4 8192,3 10598,2 13710,1 17735,1 22943,1 FIG1 Seitseman veljesta|fi|N=81840|V=21794 FIG1pts 1,4253 2,1329 3,1051 4,1012 5,820 6,770 8,538 10,485 13,409 17,325 22,292 28,234 36,173 47,142 60,118 78,99 101,71 130,58 168,46 217,37 280,29 362,23 467,18 604,15 780,12 1008,10 1302,8 1682,6 2174,5 2808,4 3628,3 4687,2 6055,2 7823,1 10107,1 13057,1 16869,1 21794,1 FIG1 Divina Commedia|it|N=97718|V=13636 FIG1pts 1,4052 2,3697 3,2361 4,1929 6,1457 7,1387 9,1042 11,826 15,739 19,645 24,569 30,378 39,302 50,232 63,193 81,153 103,121 132,91 168,74 215,54 274,42 350,33 447,25 571,19 729,14 930,11 1187,9 1515,7 1934,5 2469,4 3152,3 4023,2 5136,2 6556,1 8369,1 10682,1 13636,1 FIG2 per-book exponents: label|lang|year|N|V|a_zipf|se|a_zm|q|b|b_se|p_gof|hapax FIG2 Hamlet|en|1601|30314|4938|0.9397|0.00222|1.1406|5.789|1.9631|0.0145|0.005|0.6071 FIG2 Paradise Lost|en|1667|80169|10096|0.9471|0.00126|1.0505|2.650|1.9531|0.0214|0.475|0.5080 FIG2 Robinson Crusoe|en|1719|121569|6069|1.0242|0.00108|1.2418|4.732|1.8251|0.0199|0.925|0.3772 FIG2 Gulliver's Travels|en|1726|105375|8186|0.9995|0.00112|1.1181|2.370|1.9756|0.0281|0.980|0.4216 FIG2 Pride and Prejudice|en|1813|127858|6822|0.9944|0.00104|1.2508|7.404|1.9074|0.0333|0.140|0.3865 FIG2 Emma|en|1815|161111|7201|1.0041|0.00092|1.3078|9.421|1.7916|0.0193|0.010|0.3883 FIG2 Frankenstein|en|1818|75201|7013|0.9886|0.00135|1.0849|1.769|1.9628|0.0225|0.005|0.4299 FIG2 Moby-Dick|en|1851|216603|17371|0.9983|0.00072|1.0817|1.809|1.9580|0.0183|0.170|0.4406 FIG2 A Tale of Two Cities|en|1859|137429|9849|1.0060|0.00096|1.1499|3.255|1.8628|0.0155|0.690|0.4355 FIG2 Origin of Species|en|1859|155796|6947|0.9992|0.00094|1.1210|2.371|1.8719|0.0239|0.000|0.3258 FIG2 Great Expectations|en|1861|186711|10966|1.0229|0.00082|1.2201|4.984|1.8207|0.0139|0.205|0.4246 FIG2 Alice in Wonderland|en|1865|26776|2636|0.9519|0.00253|1.1435|4.035|1.9212|0.0395|0.030|0.4306 FIG2 Tom Sawyer|en|1876|72187|7471|0.9798|0.00137|1.1076|2.822|1.8533|0.0139|0.000|0.4841 FIG2 Huckleberry Finn|en|1884|113358|6352|1.0122|0.00111|1.2191|4.768|1.8527|0.0284|0.115|0.4260 FIG2 Sherlock Holmes|en|1892|105286|7946|1.0023|0.00112|1.1781|4.164|1.9227|0.0263|0.685|0.4437 FIG2 Dracula|en|1897|162259|9407|1.0176|0.00089|1.2351|5.718|1.8744|0.0282|0.125|0.4525 FIG2 Heart of Darkness|en|1899|38774|5455|0.9684|0.00193|1.0674|1.860|1.9763|0.0239|0.125|0.5252 FIG2 Ulysses|en|1922|264493|29738|0.9873|0.00062|1.0491|1.437|1.9526|0.0131|0.225|0.5446 FIG2 The Great Gatsby|en|1925|48676|5938|0.9640|0.00171|1.1008|3.096|1.8239|0.0112|0.020|0.5312 FIG2 L'Avare|fr|1668|22150|3376|0.9474|0.00270|1.1687|5.554|1.9131|0.0166|0.115|0.5815 FIG2 Candide|fr|1759|34450|5829|0.9369|0.00204|1.0721|3.409|1.9702|0.0198|0.960|0.5764 FIG2 Les trois mousquetaires|fr|1844|223883|15329|1.0006|0.00072|1.1978|6.328|1.8282|0.0137|0.020|0.4734 FIG2 Madame Bovary|fr|1857|112892|14140|0.9718|0.00102|1.0845|2.956|1.9628|0.0173|0.635|0.5265 FIG2 Les Fleurs du Mal|fr|1857|24855|6166|0.8975|0.00242|0.9591|1.356|2.0948|0.0150|0.135|0.6351 FIG2 Les miserables I|fr|1862|111887|12899|0.9752|0.00104|1.1139|3.887|1.8944|0.0117|0.430|0.5374 FIG2 Vingt mille lieues|fr|1870|140766|15750|0.9735|0.00091|1.0831|2.922|1.9984|0.0239|0.745|0.5112 FIG2 Du cote de chez Swann|fr|1913|168591|16613|0.9943|0.00082|1.1530|4.746|1.8984|0.0145|0.840|0.5306 FIG2 Werther I|de|1774|17734|4042|0.9053|0.00299|1.0221|2.768|2.0780|0.0182|0.000|0.6455 FIG2 Faust I|de|1808|30693|6308|0.9062|0.00217|1.0523|4.563|1.9797|0.0206|0.195|0.6308 FIG2 Also sprach Zarathustra|de|1883|84322|9450|0.9575|0.00124|1.1433|6.171|1.8268|0.0089|0.135|0.5365 FIG2 Effi Briest|de|1895|95381|11192|0.9759|0.00114|1.1901|7.443|1.8644|0.0130|0.015|0.5982 FIG2 Die Verwandlung|de|1915|19230|3790|0.9107|0.00289|1.0789|4.518|1.9314|0.0251|0.095|0.6264 FIG2 Lazarillo de Tormes|es|1554|20096|3724|0.9562|0.00280|1.0424|1.438|1.9837|0.0171|0.105|0.6114 FIG2 Don Quijote|es|1605|383633|22943|1.0446|0.00053|1.1513|2.190|1.7708|0.0072|0.000|0.4860 FIG2 Marianela|es|1878|50991|8224|0.9699|0.00161|1.0572|1.752|1.9690|0.0113|0.970|0.5951 FIG2 Divina Commedia|it|1320|97718|13636|0.9593|0.00111|1.0596|2.632|1.9230|0.0083|0.225|0.5843 FIG2 Pinocchio|it|1883|41536|6220|0.9340|0.00185|1.0411|2.512|1.9284|0.0174|0.030|0.5312 FIG2 Bras Cubas|pt|1881|63347|10138|0.9632|0.00142|1.0391|1.580|2.0120|0.0154|0.225|0.5630 FIG2 Dom Casmurro|pt|1899|67239|8855|0.9709|0.00139|1.0811|2.470|1.9280|0.0147|0.105|0.5366 FIG2 Camera Obscura|nl|1839|161385|18875|0.9903|0.00083|1.1099|3.221|1.9021|0.0069|0.120|0.5682 FIG2 Max Havelaar|nl|1860|125009|13814|0.9794|0.00097|1.1276|4.303|1.9340|0.0183|0.860|0.5366 FIG2 De Bello Gallico I-IV|la|-50|20542|5688|0.8102|0.00283|0.8814|2.385|2.0813|0.0233|0.205|0.6079 FIG2 Aeneis|la|-19|63791|16628|0.7763|0.00151|0.7835|0.231|2.4854|0.0651|0.660|0.5563 FIG2 Kalevala|fi|1849|67782|20159|0.7571|0.00147|0.8311|6.111|2.1088|0.0127|0.000|0.6089 FIG2 Seitseman veljesta|fi|1870|81840|21794|0.8591|0.00121|0.8560|-0.069|2.1116|0.0081|0.695|0.6428 FIG2 Hemsoborna|sv|1887|43433|8492|0.9511|0.00174|1.0387|1.940|2.0886|0.0127|0.000|0.6518 FIG2 Tine|da|1889|47640|5953|0.9632|0.00173|1.1709|5.928|1.8660|0.0118|0.860|0.5607 FIG2 Az uj foldesur I|hu|1863|26002|8604|0.8768|0.00231|0.8560|-0.314|2.0202|0.0365|0.930|0.7479 FIG2 Oidipous Tyrannos (mod. Greek trans.)|el|-429|13889|3663|0.8781|0.00346|1.0093|3.526|2.1475|0.0205|0.000|0.6759 FIG2 Robinsono Kruso (Esperanto trans.)|eo|1719|21352|4157|0.9470|0.00269|1.0067|0.960|1.9975|0.0164|0.750|0.6074 FIG3 local slope against rank (binned), for three books FIG3 Moby-Dick 1.4,1.134 2.8,0.340 5.6,0.893 10.9,0.834 21.0,0.898 39.7,0.950 73.6,0.968 135.7,1.090 250.0,1.195 460.4,1.103 847.4,1.037 1559.9,1.014 2871.8,1.114 5286.8,1.236 9732.1,1.377 FIG3 Don Quijote 1.6,0.139 3.5,0.932 6.6,0.882 12.2,1.061 23.1,1.005 44.0,1.165 82.9,1.100 155.8,0.976 292.5,1.015 548.3,1.105 1027.4,1.175 1925.3,1.206 3607.0,1.291 6756.2,1.335 12654.6,1.269 FIG3 Seitseman veljesta 1.4,1.678 3.0,0.482 6.2,0.766 12.2,0.660 22.9,0.811 43.1,0.849 81.0,0.933 152.2,0.902 284.5,0.895 531.3,0.855 992.1,0.861 1852.3,0.900 3458.3,0.929 6456.7,1.007 12055.3,0.399 FIG4 Moby-Dick observed against both fitted forms, at log-spaced ranks FIG4 params a_zipf=0.9983 a_zm=1.0817 q=1.809 N=216603 V=17371 FIG4obs 1,14535 2,6624 3,6447 4,4739 5,4627 6,4184 8,2532 10,1993 12,1778 15,1723 19,1396 24,1090 30,876 38,648 47,589 60,445 75,347 94,298 118,239 148,185 185,136 232,101 292,80 366,63 459,48 577,37 723,30 908,24 1139,18 1430,15 1794,12 2251,9 2825,7 3545,6 4449,4 5582,3 7005,2 8791,2 11031,1 13843,1 17371,1 FIG4zipf 1,20782.28 2,10403.59 3,6940.59 4,5208.03 5,4168.03 6,3474.45 8,2607.14 10,2086.51 12,1739.31 15,1391.98 19,1099.38 24,870.70 30,696.83 38,550.35 47,445.13 60,348.83 75,279.17 94,222.83 118,177.58 148,141.64 185,113.35 232,90.43 292,71.87 366,57.36 459,45.76 577,36.42 723,29.07 908,23.16 1139,18.47 1430,14.72 1794,11.74 2251,9.36 2825,7.46 3545,5.95 4449,4.74 5582,3.78 7005,3.01 8791,2.40 11031,1.91 13843,1.53 17371,1.22 FIG4zm 1,11992.36 2,8626.35 3,6703.54 4,5464.48 5,4601.79 6,3967.80 8,3100.45 10,2536.58 12,2141.63 15,1731.36 19,1374.36 24,1088.77 30,868.44 38,681.31 47,546.50 60,423.31 75,334.65 94,263.48 118,206.89 148,162.46 185,127.96 232,100.38 292,78.40 366,61.49 459,48.18 577,37.65 723,29.52 908,23.08 1139,18.07 1430,14.13 1794,11.06 2251,8.66 2825,6.77 3545,5.30 4449,4.14 5582,3.24 7005,2.54 8791,1.98 11031,1.55 13843,1.21 17371,0.95 FIG5 validation convergence, from VALIDATION C at b_true = 1.95 FIG5 250 1.95076 0.06187 0.06018 +0.00076 FIG5 1000 1.94768 0.02929 0.02999 -0.00232 FIG5 4000 1.94877 0.01400 0.01501 -0.00123 FIG5 16000 1.95006 0.00713 0.00752 +0.00006 FIG5 64000 1.95010 0.00386 0.00376 +0.00010 FIG5b tokeniser sensitivity and half-to-half scatter FIG5b sigma_emp 0.00436 FIG5b halfdiff 0.0059 -0.0128 0.0035 0.0049 0.0007 -0.0008 -0.0094 0.0013 -0.0009 0.0036 -0.0030 -0.0055 -0.0005 -0.0068 0.0015 0.0012 -0.0012 -0.0002 0.0022 -0.0098 -0.0006 0.0044 0.0035 -0.0040 -0.0063 -0.0043 -0.0048 -0.0044 -0.0132 0.0141 -0.0047 -0.0117 -0.0011 0.0008 -0.0105 -0.0028 0.0018 0.0006 -0.0029 0.0004 0.0112 0.0075 -0.0058 -0.0105 0.0078 -0.0029 -0.0013 -0.0089 0.0033 -0.0133 ============================================================================== SUMMARY. The headline numbers ============================================================================== books : 50 written languages : 14 tokens counted : 4,714,004 word types counted : 496,852 rank exponent a, all books : 0.9549, SD 0.0597 rank exponent a, English books : 0.9899, SD 0.0252 range of a across the corpus : 0.7571 to 1.0446 measured one-book repeatability : 0.0044 spread of a in units of that : 14 frequency exponent b, all books : 1.951, SD 0.117 books whose pure power law survives GOF : 34 of 50 books where Zipf-Mandelbrot wins on AIC : 50 of 50 Moby-Dick b, club : 1.958 +/- 0.018 Moby-Dick b, Clauset et al. 2009 : 1.95 +/- 0.02 total runtime : 611.6 s ============================================================================== END OF OUTPUT ==============================================================================