============================================================================== HOW MANY REASONABLE CHOICES DOES IT TAKE TO FIND A RESULT THAT IS NOT THERE Science Journaling Club | Volume 2, Issue 2, Winter 2026 Original research: a Monte Carlo study of researcher degrees of freedom ============================================================================== Python : 3.12.3 numpy : 2.4.2 platform : Windows-11-10.0.26200-SP0 MASTER SEED : 20260214 (every stream spawned from it via SeedSequence) nominal alpha : 0.050 two-sided group size : 40 per group at full enrolment peek schedule : 20, 25, 30, 35, 40 per group outcomes : 3, pairwise correlation r = 0.50 outlier rule : drop |z| > 2.0 SD of the sample being analysed studies (pool): 50000, shared by all 32 combinations NOTE ON WHAT IS BEING SIMULATED: there are no participants. Both groups are drawn from the same distribution, so every significant result counted below is false by construction. Nothing here was observed in anything. ============================================================================== VALIDATION 1. The club's t test against scipy.stats.ttest_ind ============================================================================== First five studies, identical data, both routines: study club t scipy t diff club p scipy p diff 0 -0.27066798 -0.27066798 +0.00e+00 0.787361352 0.787361352 +1.23e-14 1 -0.29872287 -0.29872287 +0.00e+00 0.765945912 0.765945912 +3.33e-16 2 1.52692713 1.52692713 -2.22e-16 0.130824195 0.130824195 +2.55e-15 3 1.33372508 1.33372508 +0.00e+00 0.186175542 0.186175542 +2.66e-15 4 -0.41320386 -0.41320386 +5.55e-17 0.680591027 0.680591027 -2.55e-15 Over all 20000 studies: max |t_club - t_scipy | = 1.332e-15 max |p_club - p_scipy | = 1.193e-11 max relative p error = 1.193e-11 disagreements at alpha = 0 VERDICT: agreement to better than 1e-9. ============================================================================== VALIDATION 2. The club's ANCOVA against an independent LAPACK solve ============================================================================== The club's covariate-adjusted test solves the normal equations directly. Here it is checked against numpy.linalg.lstsq, which is LAPACK's SVD-based least-squares driver and shares no code with it, with the tail probability taken from scipy.stats.t rather than from this file. (statsmodels is installed in this environment but will not import against numpy 2.4.2, so the comparison is made this way instead.) study club t_group LAPACK t diff club p scipy p diff 0 0.39405005 0.39405005 +5.55e-17 0.694633263 0.694633263 -1.08e-14 1 0.32642999 0.32642999 +5.55e-17 0.744984651 0.744984651 -5.88e-15 2 -1.49747335 -1.49747335 +1.55e-15 0.138358392 0.138358392 -3.06e-14 3 -1.34073152 -1.34073152 -6.66e-16 0.183949716 0.183949716 -3.02e-14 4 0.39666982 0.39666982 -5.55e-17 0.692708203 0.692708203 -1.49e-14 Over 500 studies: max |t diff| = 3.553e-15, max |p diff| = 6.716e-13 degrees of freedom used by both: 77 VERDICT: agreement to better than 1e-9. ============================================================================== VALIDATION 2b. A t test small enough to check by hand ============================================================================== group A: [4.0, 7.0, 5.0, 6.0, 8.0] mean 6.0000 variance 2.5000 group B: [6.0, 9.0, 7.0, 10.0, 8.0] mean 8.0000 variance 2.5000 pooled variance = (4*2.5000 + 4*2.5000)/8 = 2.5000 standard error = sqrt(2.5000 * (1/5 + 1/5)) = 1.000000 t = (6.0 - 8.0) / 1.000000 = -2.000000 df = 8 club p = 0.080516238 scipy t, p = -2.000000, 0.080516238 difference = +0.00e+00, +2.78e-16 ============================================================================== THE MAIN POOL ============================================================================== Simulating 50000 studies and evaluating all 180 analysis paths in each. Every path is a defensible analysis of the same null data. ... 10000 / 50000 studies ... 20000 / 50000 studies ... 30000 / 50000 studies ... 40000 / 50000 studies ... 50000 / 50000 studies pool complete in 62.8 s ============================================================================== VALIDATION 3. The baseline. No researcher degrees of freedom. ============================================================================== One outcome measure, one test, at the planned sample size of 40 per group. Nothing is chosen after seeing the data. This is the check on everything else in this file: if it does not come out at 0.050, nothing below means anything. significant studies : 2522 of 50000 club false positive rate: 0.05044 standard error : 0.00098 95% Wilson interval : [0.04856, 0.05239] accepted (nominal) value: 0.05000 difference : +0.00044 distance in SE : +0.45 sigma interval covers 0.05 : yes VERDICT: baseline recovers the nominal level. Proceed. Further check on the same numbers: under the null the p values must be uniform on (0,1). Deciles of the baseline p values, expected 5000 each: 4963 4973 4917 4992 5020 5089 5021 5058 5056 4911 chi-square on 9 df = 6.45 (5% critical value 16.92, 1% is 21.67) ============================================================================== VALIDATION 4. Optional stopping against Armitage, McPherson & Rowe (1969) ============================================================================== Equally spaced looks at accumulating data, 25 new participants per group between looks, stop at the first significant result. This is the setting the 1969 table describes, and it is NOT the peek schedule used in the main grid, which adds only 20 participants per group in total. Studies: 200000. Look-by-look critical values from the club's own t quantile. 200000 studies in 5.6 s looks club rate SE 95% interval published diff sigma 1 0.04952 0.00049 [0.04857,0.05047] 0.050 -0.0005 -1.00 2 0.08228 0.00061 [0.08109,0.08350] 0.083 -0.0007 -1.16 3 0.10731 0.00069 [0.10597,0.10868] 0.107 +0.0003 +0.46 4 0.12657 0.00074 [0.12512,0.12803] 0.126 +0.0006 +0.77 5 0.14200 0.00078 [0.14048,0.14354] 0.142 +0.0000 +0.00 10 0.19424 0.00088 [0.19251,0.19598] 0.193 +0.0012 +1.40 20 0.24858 0.00097 [0.24669,0.25048] 0.246 +0.0026 +2.67 Full curve, looks 1 to 20: K=1 0.04952 K=2 0.08228 K=3 0.10731 K=4 0.12657 K=5 0.14200 K=6 0.15548 K=7 0.16679 K=8 0.17679 K=9 0.18595 K=10 0.19424 K=11 0.20136 K=12 0.20852 K=13 0.21497 K=14 0.22060 K=15 0.22614 K=16 0.23095 K=17 0.23589 K=18 0.24047 K=19 0.24458 K=20 0.24858 largest absolute deviation from the published table: 0.0026 NOTE: the published values are rounded to three decimal places and are the large-sample limit. At 25 observations per group per look the discrete t test is close to that limit but not identical to it, so an agreement of a few thousandths is the most this comparison can show. ============================================================================== VALIDATION 5. Reproducing Table 1 of Simmons, Nelson & Simonsohn (2011) ============================================================================== That paper is the reason this study exists. Its Table 1 gives false positive rates for four researcher degrees of freedom, from 15,000 simulations each, at three significance levels. Three of the four can be built out of the pieces already in this file, so we built them and ran 120000 simulations of each. Situation D needs a three-condition design, which our model does not have, so it is not attempted. Every operationalisation is taken from the note under their table, not from our own reading of the text. Gender is a fair coin per observation. 120000 simulations per row in 37.6 s Their table rests on 15,000 simulations per cell, so it carries Monte Carlo error of its own. The sigma column below is the difference over the standard error of the DIFFERENCE, which combines both runs. row situation alpha club published diff their SE sigma SIM A two dependent variables (r = .50) 0.10 0.18649 0.178 +0.0085 0.00312 +2.56 SIM 0.05 0.09898 0.095 +0.0040 0.00239 +1.57 SIM 0.01 0.02077 0.022 -0.0012 0.00120 -0.97 SIM B addition of 10 more observations per cell 0.10 0.14922 0.145 +0.0042 0.00287 +1.38 SIM 0.05 0.07761 0.077 +0.0006 0.00218 +0.26 SIM 0.01 0.01630 0.016 +0.0003 0.00102 +0.28 SIM C controlling for gender or its interaction 0.10 0.22368 0.216 +0.0077 0.00336 +2.15 SIM 0.05 0.12054 0.117 +0.0035 0.00262 +1.27 SIM 0.01 0.02617 0.027 -0.0008 0.00132 -0.59 SIM AB combine A and B 0.10 0.26775 0.260 +0.0077 0.00358 +2.04 SIM 0.05 0.14808 0.144 +0.0041 0.00287 +1.34 SIM 0.01 0.03425 0.033 +0.0013 0.00146 +0.81 SIM ABC combine A, B and C 0.10 0.51498 0.509 +0.0060 0.00408 +1.38 SIM 0.05 0.31971 0.309 +0.0107 0.00377 +2.67 SIM 0.01 0.08618 0.084 +0.0022 0.00226 +0.91 largest deviation: 2.67 sigma, at ABC at alpha 0.05. The published figures are also rounded to a tenth of a percentage point, worth up to 0.0005 on its own, so an agreement closer than that cannot be demonstrated by this comparison however long we run. VERDICT: our implementation reproduces the published table. One ambiguity we could not resolve from the published description. In the model carrying a gender interaction, the condition coefficient depends on how gender is coded. With gender as a 0/1 dummy it is the condition effect among the gender coded 0; with gender centred it is the average condition effect. The rows above use the 0/1 dummy, which is what regression software does by default. Repeating Situation C with gender centred instead: SIMC alpha 0.10 centred 0.20213 dummy 0.22368 published 0.216 SIMC alpha 0.05 centred 0.10518 dummy 0.12054 published 0.117 SIMC alpha 0.01 centred 0.02122 dummy 0.02617 published 0.027 Replicating a paper about analytic flexibility required us to make an analytic choice the paper did not pin down. We report both. ============================================================================== THE GRID. All 32 combinations of five researcher degrees of freedom ============================================================================== S = optional stopping, M = three outcome measures, G = subgroup on an irrelevant split, C = covariate inclusion, O = outlier exclusion at 2 SD. Every combination takes the smallest p value among the analyses it permits. k_eff is the number of independent 5% tests that would give the same rate. combo freedoms paths k_sig FPR SE 95% interval k_eff SE (none) baseline 1 2522 0.05044 0.00098 [0.04856,0.05239] 1.01 0.02 C C 2 2742 0.05484 0.00102 [0.05288,0.05687] 1.10 0.02 G G 3 5827 0.11654 0.00143 [0.11376,0.11938] 2.42 0.03 M M 3 6428 0.12856 0.00150 [0.12565,0.13152] 2.68 0.03 O O 2 3681 0.07362 0.00117 [0.07136,0.07594] 1.49 0.02 S S 5 5697 0.11394 0.00142 [0.11118,0.11675] 2.36 0.03 CO C+O 4 4013 0.08026 0.00122 [0.07791,0.08267] 1.63 0.03 GC G+C 6 6498 0.12996 0.00150 [0.12704,0.13294] 2.71 0.03 GO G+O 6 8272 0.16544 0.00166 [0.16221,0.16872] 3.53 0.04 MC M+C 6 6941 0.13882 0.00155 [0.13582,0.14188] 2.91 0.04 MG M+G 9 13826 0.27652 0.00200 [0.27262,0.28046] 6.31 0.05 MO M+O 6 9285 0.18570 0.00174 [0.18232,0.18913] 4.00 0.04 SC S+C 10 6230 0.12460 0.00148 [0.12173,0.12752] 2.59 0.03 SG S+G 15 12344 0.24688 0.00193 [0.24312,0.25068] 5.53 0.05 SM S+M 15 13496 0.26992 0.00199 [0.26605,0.27383] 6.13 0.05 SO S+O 10 8188 0.16376 0.00165 [0.16054,0.16703] 3.49 0.04 GCO G+C+O 12 9229 0.18458 0.00173 [0.18120,0.18800] 3.98 0.04 MCO M+C+O 12 10009 0.20018 0.00179 [0.19670,0.20371] 4.35 0.04 MGC M+G+C 18 15106 0.30212 0.00205 [0.29811,0.30616] 7.01 0.06 MGO M+G+O 18 19150 0.38300 0.00217 [0.37875,0.38727] 9.41 0.07 SCO S+C+O 20 8928 0.17856 0.00171 [0.17523,0.18194] 3.83 0.04 SGC S+G+C 30 13763 0.27526 0.00200 [0.27136,0.27919] 6.28 0.05 SGO S+G+O 30 16796 0.33592 0.00211 [0.33179,0.34007] 7.98 0.06 SMC S+M+C 30 14608 0.29216 0.00203 [0.28819,0.29616] 6.74 0.06 SMG S+M+G 45 26031 0.52062 0.00223 [0.51624,0.52500] 14.33 0.09 SMO S+M+O 30 18799 0.37598 0.00217 [0.37174,0.38024] 9.19 0.07 MGCO M+G+C+O 36 20768 0.41536 0.00220 [0.41105,0.41969] 10.46 0.07 SGCO S+G+C+O 60 18587 0.37174 0.00216 [0.36751,0.37599] 9.06 0.07 SMCO S+M+C+O 60 20207 0.40414 0.00219 [0.39985,0.40845] 10.09 0.07 SMGC S+M+G+C 90 28260 0.56520 0.00222 [0.56085,0.56954] 16.24 0.10 SMGO S+M+G+O 90 33082 0.66164 0.00212 [0.65748,0.66577] 21.13 0.12 SMGCO S+M+G+C+O 180 35331 0.70662 0.00204 [0.70261,0.71059] 23.91 0.14 ONE AT A TIME, in order of damage: M three outcome measures 0.12856 +/- 0.00150 (2.6x nominal, k_eff 2.68) G subgroup on irrelevant split 0.11654 +/- 0.00143 (2.3x nominal, k_eff 2.42) S optional stopping (5 looks) 0.11394 +/- 0.00142 (2.3x nominal, k_eff 2.36) O outlier exclusion at 2 SD 0.07362 +/- 0.00117 (1.5x nominal, k_eff 1.49) C covariate inclusion 0.05484 +/- 0.00102 (1.1x nominal, k_eff 1.10) CUMULATIVE, adopting freedoms in the order S, M, G, C, O: after 0 freedom(s) none paths 1 FPR 0.05044 +/- 0.00098 step +0.00000 k_eff 1.01 after 1 freedom(s) S paths 5 FPR 0.11394 +/- 0.00142 step +0.06350 k_eff 2.36 after 2 freedom(s) SM paths 15 FPR 0.26992 +/- 0.00199 step +0.15598 k_eff 6.13 after 3 freedom(s) SMG paths 45 FPR 0.52062 +/- 0.00223 step +0.25070 k_eff 14.33 after 4 freedom(s) SMGC paths 90 FPR 0.56520 +/- 0.00222 step +0.04458 k_eff 16.24 after 5 freedom(s) SMGCO paths 180 FPR 0.70662 +/- 0.00204 step +0.14142 k_eff 23.91 The order matters for the intermediate rows and not for the last one. Adopting in the reverse order O, C, G, M, S: after 0 freedom(s) none paths 1 FPR 0.05044 step +0.00000 after 1 freedom(s) O paths 2 FPR 0.07362 step +0.02318 after 2 freedom(s) CO paths 4 FPR 0.08026 step +0.00664 after 3 freedom(s) GCO paths 12 FPR 0.18458 step +0.10432 after 4 freedom(s) MGCO paths 36 FPR 0.41536 step +0.23078 after 5 freedom(s) SMGCO paths 180 FPR 0.70662 step +0.29126 HEADLINE: with all five freedoms in play, 35331 of 50000 studies of pure noise produced a significant result. That is 0.7066, or 70.7%, against a nominal 5%. The inflation factor is 14.13. The 180 analysis paths behave like 23.91 independent tests, which is 13.3% of their nominal count: the paths overlap heavily because they share data. ============================================================================== EFFECTIVE INDEPENDENT TESTS: nominal paths against k_eff ============================================================================== combo paths k_eff k_eff/paths redundancy (none) 1 1.01 1.009 0.99 paths per effective test C 2 1.10 0.550 1.82 paths per effective test G 3 2.42 0.805 1.24 paths per effective test M 3 2.68 0.894 1.12 paths per effective test O 2 1.49 0.745 1.34 paths per effective test S 5 2.36 0.472 2.12 paths per effective test CO 4 1.63 0.408 2.45 paths per effective test GC 6 2.71 0.452 2.21 paths per effective test GO 6 3.53 0.588 1.70 paths per effective test MC 6 2.91 0.486 2.06 paths per effective test MG 9 6.31 0.701 1.43 paths per effective test MO 6 4.00 0.667 1.50 paths per effective test SC 10 2.59 0.259 3.85 paths per effective test SG 15 5.53 0.369 2.71 paths per effective test SM 15 6.13 0.409 2.45 paths per effective test SO 10 3.49 0.349 2.87 paths per effective test GCO 12 3.98 0.332 3.02 paths per effective test MCO 12 4.35 0.363 2.76 paths per effective test MGC 18 7.01 0.390 2.57 paths per effective test MGO 18 9.41 0.523 1.91 paths per effective test SCO 20 3.83 0.192 5.22 paths per effective test SGC 30 6.28 0.209 4.78 paths per effective test SGO 30 7.98 0.266 3.76 paths per effective test SMC 30 6.74 0.225 4.45 paths per effective test SMG 45 14.33 0.319 3.14 paths per effective test SMO 30 9.19 0.306 3.26 paths per effective test MGCO 36 10.46 0.291 3.44 paths per effective test SGCO 60 9.06 0.151 6.62 paths per effective test SMCO 60 10.09 0.168 5.94 paths per effective test SMGC 90 16.24 0.180 5.54 paths per effective test SMGO 90 21.13 0.235 4.26 paths per effective test SMGCO 180 23.91 0.133 7.53 paths per effective test ============================================================================== CONVERGENCE ============================================================================== Running estimate as studies accumulate, for the baseline and for the full combination. Monte Carlo error falls as 1/sqrt(n) and nothing else changes, so a trace that is still wandering at the right-hand edge would mean the run was too short. CONVERGENCE TABLE (trials, baseline, +/-2SE, S+M+G, all five) CONV 100 0.03000 -0.00412 0.06412 0.50000 0.63000 CONV 111 0.02703 -0.00376 0.05781 0.51351 0.63964 CONV 123 0.04065 0.00504 0.07626 0.50407 0.62602 CONV 137 0.03650 0.00445 0.06854 0.51825 0.65693 CONV 152 0.03947 0.00789 0.07106 0.52632 0.66447 CONV 169 0.04142 0.01076 0.07208 0.50888 0.66272 CONV 188 0.04255 0.01311 0.07200 0.52660 0.68085 CONV 209 0.05263 0.02174 0.08352 0.52632 0.68900 CONV 232 0.05172 0.02264 0.08080 0.51724 0.69397 CONV 258 0.06977 0.03805 0.10149 0.51550 0.69380 CONV 287 0.06969 0.03963 0.09975 0.54355 0.71080 CONV 319 0.06270 0.03555 0.08984 0.54545 0.69906 CONV 354 0.06215 0.03648 0.08781 0.53390 0.69209 CONV 393 0.06107 0.03691 0.08523 0.53181 0.70483 CONV 437 0.05950 0.03686 0.08213 0.53089 0.70709 CONV 485 0.06392 0.04170 0.08613 0.54227 0.71546 CONV 539 0.06308 0.04214 0.08402 0.54174 0.71243 CONV 599 0.05843 0.03926 0.07760 0.53756 0.71119 CONV 666 0.05405 0.03653 0.07158 0.54505 0.71471 CONV 740 0.05135 0.03512 0.06758 0.53243 0.70270 CONV 822 0.05474 0.03888 0.07061 0.53771 0.70925 CONV 913 0.05476 0.03970 0.06982 0.53341 0.70646 CONV 1015 0.05419 0.03998 0.06840 0.53399 0.70936 CONV 1128 0.05319 0.03983 0.06656 0.53191 0.70656 CONV 1253 0.04948 0.03723 0.06173 0.53631 0.70950 CONV 1392 0.04885 0.03730 0.06041 0.53305 0.70905 CONV 1547 0.04460 0.03411 0.05510 0.53523 0.71041 CONV 1718 0.04424 0.03432 0.05416 0.54191 0.71478 CONV 1909 0.04243 0.03320 0.05166 0.54217 0.71451 CONV 2121 0.04290 0.03410 0.05170 0.54361 0.71476 CONV 2357 0.04328 0.03489 0.05166 0.54306 0.71616 CONV 2619 0.04429 0.03625 0.05233 0.54181 0.71554 CONV 2910 0.04570 0.03796 0.05345 0.54124 0.71581 CONV 3233 0.04702 0.03957 0.05446 0.53882 0.71482 CONV 3592 0.04788 0.04076 0.05501 0.54232 0.71743 CONV 3991 0.04786 0.04110 0.05462 0.53921 0.71135 CONV 4434 0.04939 0.04288 0.05590 0.53473 0.70478 CONV 4927 0.04851 0.04239 0.05463 0.52973 0.70428 CONV 5474 0.04932 0.04347 0.05518 0.52758 0.70168 CONV 6082 0.04982 0.04424 0.05540 0.52696 0.70585 CONV 6758 0.04927 0.04401 0.05454 0.52826 0.70642 CONV 7509 0.05021 0.04517 0.05525 0.52777 0.70689 CONV 8343 0.05022 0.04544 0.05500 0.52883 0.70778 CONV 9269 0.05092 0.04636 0.05549 0.52821 0.70795 CONV 10299 0.04962 0.04534 0.05390 0.52607 0.70745 CONV 11443 0.04990 0.04583 0.05397 0.52600 0.70812 CONV 12714 0.04939 0.04555 0.05324 0.52611 0.70914 CONV 14126 0.04934 0.04570 0.05299 0.52506 0.70834 CONV 15695 0.04855 0.04512 0.05198 0.52361 0.70800 CONV 17439 0.04851 0.04526 0.05177 0.52411 0.70761 CONV 19376 0.04862 0.04553 0.05171 0.52276 0.70618 CONV 21528 0.04915 0.04620 0.05209 0.52313 0.70754 CONV 23920 0.04937 0.04657 0.05217 0.52354 0.70778 CONV 26577 0.04933 0.04667 0.05199 0.52414 0.70730 CONV 29529 0.05012 0.04758 0.05266 0.52386 0.70768 CONV 32809 0.04986 0.04746 0.05227 0.52126 0.70526 CONV 36453 0.05009 0.04781 0.05238 0.52138 0.70686 CONV 40502 0.05061 0.04844 0.05279 0.52022 0.70609 CONV 45001 0.05020 0.04814 0.05226 0.52017 0.70587 CONV 50000 0.05044 0.04848 0.05240 0.52062 0.70662 last 10000 studies only: baseline 0.05030, all five 0.70990 first 10000 studies only: baseline 0.04970, all five 0.70830 DEPTH. For each study, the first point in the order S, M, G, C, O at which it became significant. 'never' means no permitted path reached p < 0.05 even with all five freedoms. DEPTH 0 0 (honest test) 2522 5.04% DEPTH 1 1 (peeking) 3175 6.35% DEPTH 2 2 (+ outcomes) 7799 15.60% DEPTH 3 3 (+ subgroups) 12535 25.07% DEPTH 4 4 (+ covariate) 2229 4.46% DEPTH 5 5 (+ outliers) 7071 14.14% DEPTH - never 14669 29.34% ============================================================================== SENSITIVITY. Where a different modelling choice changes the answer ============================================================================== Each cell below is an independent run of 30000 studies with its own stream. A. Correlation between the three outcome measures. Choosing among outcomes costs least when the outcomes agree with each other. r FPR(M alone) SE k_eff FPR(all five) SE SENSR 0.00 0.14197 0.00202 2.99 0.75247 0.00249 SENSR 0.30 0.13863 0.00200 2.91 0.74007 0.00253 SENSR 0.50 0.12437 0.00191 2.59 0.70630 0.00263 SENSR 0.80 0.09857 0.00172 2.02 0.61320 0.00281 The main grid uses r = 0.5. At r = 0 the three measures are three independent tests; at r = 0.8 they are nearly one. The whole result moves with a parameter nobody measured. B. The outlier threshold. 2 SD, 2.5 SD and 3 SD are all rules people defend in print. threshold mean dropped FPR(O alone) SE k_eff FPR(all five) SENST 2.0 4.55% 0.07257 0.00150 1.47 0.70320 SENST 2.5 1.24% 0.06543 0.00143 1.32 0.61877 SENST 3.0 0.27% 0.05023 0.00126 1.00 0.57340 A stricter rule drops fewer points, so it moves the p value less, so it buys less. The 2 SD rule in the main grid is the most generous of the three and our headline number depends on that choice. C. The peek schedule. The main grid peeks four extra times while adding only 20 participants per group. Armitage's setting doubles, triples and quadruples the sample instead. main grid, S alone, 5 looks at n = 20..40 : 0.11394 equally spaced, 5 looks at n = 25..125 : 0.14200 ratio : 1.25 The same word, 'we peeked five times', covers both. They are not the same act and they do not cost the same. ============================================================================== SUMMARY TABLE FOR THE ARTICLE ============================================================================== TAB combo paths k_sig FPR SE lo hi k_eff keff_se inflation TAB - 1 2522 0.05044 0.00098 0.04856 0.05239 1.01 0.02 1.01 TAB C 2 2742 0.05484 0.00102 0.05288 0.05687 1.10 0.02 1.10 TAB G 3 5827 0.11654 0.00143 0.11376 0.11938 2.42 0.03 2.33 TAB M 3 6428 0.12856 0.00150 0.12565 0.13152 2.68 0.03 2.57 TAB O 2 3681 0.07362 0.00117 0.07136 0.07594 1.49 0.02 1.47 TAB S 5 5697 0.11394 0.00142 0.11118 0.11675 2.36 0.03 2.28 TAB CO 4 4013 0.08026 0.00122 0.07791 0.08267 1.63 0.03 1.61 TAB GC 6 6498 0.12996 0.00150 0.12704 0.13294 2.71 0.03 2.60 TAB GO 6 8272 0.16544 0.00166 0.16221 0.16872 3.53 0.04 3.31 TAB MC 6 6941 0.13882 0.00155 0.13582 0.14188 2.91 0.04 2.78 TAB MG 9 13826 0.27652 0.00200 0.27262 0.28046 6.31 0.05 5.53 TAB MO 6 9285 0.18570 0.00174 0.18232 0.18913 4.00 0.04 3.71 TAB SC 10 6230 0.12460 0.00148 0.12173 0.12752 2.59 0.03 2.49 TAB SG 15 12344 0.24688 0.00193 0.24312 0.25068 5.53 0.05 4.94 TAB SM 15 13496 0.26992 0.00199 0.26605 0.27383 6.13 0.05 5.40 TAB SO 10 8188 0.16376 0.00165 0.16054 0.16703 3.49 0.04 3.28 TAB GCO 12 9229 0.18458 0.00173 0.18120 0.18800 3.98 0.04 3.69 TAB MCO 12 10009 0.20018 0.00179 0.19670 0.20371 4.35 0.04 4.00 TAB MGC 18 15106 0.30212 0.00205 0.29811 0.30616 7.01 0.06 6.04 TAB MGO 18 19150 0.38300 0.00217 0.37875 0.38727 9.41 0.07 7.66 TAB SCO 20 8928 0.17856 0.00171 0.17523 0.18194 3.83 0.04 3.57 TAB SGC 30 13763 0.27526 0.00200 0.27136 0.27919 6.28 0.05 5.51 TAB SGO 30 16796 0.33592 0.00211 0.33179 0.34007 7.98 0.06 6.72 TAB SMC 30 14608 0.29216 0.00203 0.28819 0.29616 6.74 0.06 5.84 TAB SMG 45 26031 0.52062 0.00223 0.51624 0.52500 14.33 0.09 10.41 TAB SMO 30 18799 0.37598 0.00217 0.37174 0.38024 9.19 0.07 7.52 TAB MGCO 36 20768 0.41536 0.00220 0.41105 0.41969 10.46 0.07 8.31 TAB SGCO 60 18587 0.37174 0.00216 0.36751 0.37599 9.06 0.07 7.43 TAB SMCO 60 20207 0.40414 0.00219 0.39985 0.40845 10.09 0.07 8.08 TAB SMGC 90 28260 0.56520 0.00222 0.56085 0.56954 16.24 0.10 11.30 TAB SMGO 90 33082 0.66164 0.00212 0.65748 0.66577 21.13 0.12 13.23 TAB SMGCO 180 35331 0.70662 0.00204 0.70261 0.71059 23.91 0.14 14.13 ARM K club published ARM 1 0.04952 0.050 ARM 2 0.08228 0.083 ARM 3 0.10731 0.107 ARM 4 0.12657 0.126 ARM 5 0.14200 0.142 ARM 6 0.15548 - ARM 7 0.16679 - ARM 8 0.17679 - ARM 9 0.18595 - ARM 10 0.19424 0.193 ARM 11 0.20136 - ARM 12 0.20852 - ARM 13 0.21497 - ARM 14 0.22060 - ARM 15 0.22614 - ARM 16 0.23095 - ARM 17 0.23589 - ARM 18 0.24047 - ARM 19 0.24458 - ARM 20 0.24858 0.246 ============================================================================== Total wall clock: 377.9 s (6.3 min) Master seed 20260214. Rerunning this file reproduces every number above. ==============================================================================