============================================================================== eDNA versus conventional survey detection: a club meta-analysis Science Journaling Club | analysis/edna-detection-review.py ============================================================================== ============================================================================== PART 0. VALIDATION OF THE POOLING CODE ============================================================================== Check 1. Recovery of a known true effect. We simulate 14 studies drawn from a random-effects world with a true mean log odds ratio of MU = 0.800 and a true between-study SD of TAU = 0.450. Each study gets its own within-study variance. If the code is right, the pooled estimate should sit near 0.800 and its 95% interval should contain it in roughly 95% of repeats. single synthetic data set, k = 14 pooled log OR 1.194 [ 0.866, 1.523] true value MU 0.800 interval contains MU: NO tau-hat 0.457 against true TAU 0.450 I-squared 62.8%, Q = 34.91 on 13 df (p = 0.0009) 4000 repeats: mean pooled estimate 0.8017 (true 0.8000, bias +0.0017) mean tau-squared 0.2045 (true 0.2025) 95% interval coverage 91.8% (nominal 95.0%) VERDICT: PASS Check 2. The tau-squared = 0 identity. When Q <= df the DerSimonian-Laird estimator truncates tau-squared to zero, and the random-effects weights 1/(v + 0) become the fixed-effect weights 1/v exactly. The two estimates and their standard errors must then be identical, not merely close. We build a data set with no real heterogeneity and check it to machine precision. Q = 0.0082 on 6 df, tau-squared = 0.000000 fixed effect 0.301070568331 (SE 0.129509648945) random effects 0.301070568331 (SE 0.129509648945) absolute differences: estimate 0.000e+00, SE 0.000e+00 VERDICT: PASS Check 3. Egger's test on symmetric data. Feed the regression a funnel that is symmetric by construction. The intercept should be statistically indistinguishable from zero. intercept +0.498 (SE 0.378), t = +1.31 on 8 df, p = 0.225 VERDICT: PASS ALL VALIDATION CHECKS: PASS ============================================================================== PART 1. THE EXTRACTED STUDIES ============================================================================== study target p(eDNA) p(conv) log OR ------------------------------------------------------------------------------ Biggs 2015 Great crested newt (Triturus crist 0.99 0.75 +3.443 Valentini 2016 Amphibian assemblage, 39 French wa 0.97 0.58 +3.153 Schmelzle 2016 Tidewater goby (Eucyclogobius newb 0.74 0.39 +1.493 Hinlo 2017 Redfin perch (Perca fluviatilis) 0.21 0.00 +2.178 Eiler 2018 Pool frog (Pelophylax lessonae) 0.38 0.40 -0.084 Rose 2019 Northern watersnake (Nerodia siped 0.44 0.74 -1.287 Akre 2019 Wood turtle (Glyptemys insculpta) 0.55 0.88 -1.792 Moss 2022 California newt (Taricha torosa) 0.65 0.69 -0.181 Quilumbaquin 2023 Amphibian assemblage, Ecuadorian A 0.42 0.17 +1.263 Li 2024 Amphibian assemblage, Zhoushan Arc 0.54 0.24 +1.313 Di Girolamo 2024 American mink (Neogale vison) 0.25 0.36 -0.523 Dougherty 2025 Alewife (Alosa pseudoharengus) 0.57 1.00 -2.457 ------------------------------------------------------------------------------ Per-study detail, with the uncertainty we used and where it came from Biggs 2015 [ref 8] target Great crested newt (Triturus cristatus) comparator torch count (median of 3 field methods) survey unit visit; n = 140; volume 0.09 L; filter none (ethanol precipitation) y = +3.443, var = 0.7115, SE = 0.843 (input type: counts) source Text: eDNA detected newts on 139 of 140 visits (99.3%); bottle traps 76%, torch counts 75%, egg searches 44%, over 35 ponds visited 4 times. Valentini 2016 [ref 7] target Amphibian assemblage, 39 French water bodies comparator single traditional visit (netting, visual, calling) survey unit visit; n = 39; volume 0.09 L; filter none (ethanol precipitation) y = +3.153, var = 0.3926, SE = 0.627 (input type: ci) source Results: 'the detection probability with eDNA metabarcoding was 0.97 (CI = 0.90-0.99) vs. 0.58 (CI = 0.50-0.63) for traditional surveys.' Schmelzle 2016 [ref 9] target Tidewater goby (Eucyclogobius newberryi) comparator seine haul survey unit sample; n = 29; volume not recorded L; filter not recorded by us y = +1.493, var = 0.3242, SE = 0.569 (input type: derived) source Abstract/summary: detection probability 0.74 for eDNA vs 0.39 for seining across 29 paired locations. No uncertainty reported in the text available to us; SE reconstructed from n = 29 sites. Hinlo 2017 [ref 10] target Redfin perch (Perca fluviatilis) comparator fyke net set survey unit site-season; n = 14; volume 12.00 L; filter 1.2 um glass fibre y = +2.178, var = 2.4416, SE = 1.563 (input type: counts) source Table 1 detection matrix, 8 sites x 2 seasons, restricted to the 14 site-seasons where both methods ran. Median of three species contrasts (carp -1.17, redfin +2.18, weatherloach +4.36 on the log odds scale). Eiler 2018 [ref 11] target Pool frog (Pelophylax lessonae) comparator calling-male count plus visual survey unit visit; n = 49; volume 0.85 L; filter 0.45 um y = -0.084, var = 0.1717, SE = 0.414 (input type: derived) source Results: 'detection probability for P. lessonae was 0.38 per sample with the eDNA method compared to 0.40 when detecting calling males and silent observed individuals'; 49 matched samples. SE reconstructed from n = 49. Rose 2019 [ref 12] target Northern watersnake (Nerodia sipedon) comparator one trap-night (30 traps) survey unit sample; n = 61; volume 0.50 L; filter 0.45 um y = -1.287, var = 1.1177, SE = 1.057 (input type: ci) source Reported per-unit detection with 95% credible intervals: eDNA 0.44 (0.15-0.75), trapping 0.74 (0.48-0.95). Median of two species contrasts; N. sipedon has the larger site count (61). Akre 2019 [ref 13] target Wood turtle (Glyptemys insculpta) comparator one-hour visual encounter survey of 1 km survey unit sample; n = 37; volume 2.00 L; filter 0.45 um cellulose nitrate y = -1.792, var = 0.9538, SE = 0.977 (input type: ci) source Reported detection probabilities with 95% CI: eDNA per sample 0.55 (0.38-0.71); visual encounter survey 0.88 (0.58-0.98); 37 stream reaches. Moss 2022 [ref 14] target California newt (Taricha torosa) comparator seine haul survey unit visit; n = 20; volume 0.25 L; filter 0.45 or 5 um y = -0.181, var = 0.3828, SE = 0.619 (input type: ci) source Results: metabarcoding p = 0.65 (95% CI 0.44-0.81), seining p = 0.69 (0.48-0.84), 20 ponds. Median of the three species with full intervals reported (red-legged frog +3.24, newt -0.18, bullfrog -0.18). Quilumbaquin 2023 [ref 15] target Amphibian assemblage, Ecuadorian Amazon comparator visual encounter survey survey unit visit; n = 36; volume 1.00 L; filter 0.45 um nitrocellulose y = +1.263, var = 0.0147, SE = 0.121 (input type: ci) source Abstract: 'eDNA detected 28 species and had a detection probability (DP) of 0.42 CI [0.40-0.45], while VES recorded 20 species with a DP of 0.17 CI [0.14-0.20].' Li 2024 [ref 16] target Amphibian assemblage, Zhoushan Archipelago comparator traditional line transect method survey unit site; n = 21; volume 2.00 L; filter 1.5 um glass fibre y = +1.313, var = 0.4528, SE = 0.673 (input type: derived) source Results: 'the mean detection probability of eDNA is 0.54, while the mean detection probability of TLTM is 0.24'; 21 islands. No uncertainty reported; SE reconstructed from n = 21 islands. Di Girolamo 2024 [ref 17] target American mink (Neogale vison) comparator one camera-trap week survey unit visit; n = 21; volume 0.50 L; filter 0.45 um nitrocellulose y = -0.523, var = 0.6643, SE = 0.815 (input type: se) source Results: camera traps rho = 0.36 (SE 0.16), eDNA rho = 0.25 (SE 0.08), 7 sites over 21 survey weeks. Dougherty 2025 [ref 18] target Alewife (Alosa pseudoharengus) comparator purse seine survey (6 sets) survey unit site-season; n = 7; volume 3.55 L; filter 1.2 um glass microfibre y = -2.457, var = 2.6413, SE = 1.625 (input type: counts) source Results: 'we observed a 51.7% detection rate with 3 false negatives out of a sample size of 7 lakes'; seining detected alewife in every lake where the species was present. Studies read and deliberately excluded from pooling: - Smart 2015 (ref 19), smooth newt, Melbourne reports per-site RANGES only (eDNA 0.29-1.0 per sample, traps 0.01-0.26 per trap) with no pooled estimate or interval. Nothing to pool. - Hunter 2015 (ref 20), Burmese python, Florida conventional effort returned zero detections in 5,935 trap-nights and zero standardised visual sightings. The odds ratio is not finite and a continuity correction would invent the comparison. - Lopes 2017 (ref 21), tropical stream anurans, Brazil eDNA detection probabilities only (no paired conventional p), but the 20 L vs 60 L contrast is used below as direct evidence on volume. - Dougherty 2016 crayfish, rusty crayfish, Wisconsin eDNA detection modelled against trap CPUE as a covariate; no detection probability estimated for trapping itself. - de Souza 2016, waterdog and musk turtle, Alabama eDNA seasonal detection only; no conventional comparator fitted. - Lyet 2021, terrestrial mammals via stream eDNA per-taxon eDNA detection given for 30-80 L samples, camera comparison expressed as taxa counts rather than a paired per-unit probability. - NCDOT RP2023-18, Roanoke logperch, North Carolina grey literature: eDNA 6/8 vs electrofishing 1/8 at historically occupied sites. Excluded because it is not peer reviewed. ============================================================================== PART 2. POOLED RANDOM-EFFECTS ESTIMATE (DerSimonian-Laird) ============================================================================== k = 12 contrasts from 12 studies random effects 0.718 [ -0.048, 1.483] as an odds ratio 2.049 [ 0.953, 4.405] z = +1.838, p = 0.0661 fixed effect 1.128 [ 0.921, 1.335] Cochran's Q 56.32 on 11 df, p = 0.00000 tau-squared 1.2008 (tau = 1.096 on the log odds scale) I-squared 80.5% H-squared 5.12 95% prediction interval [ -1.874, 3.310] (odds ratio 0.15 to 27.37) Read on the probability scale: if one conventional survey has a detection probability of 0.50, the pooled odds ratio puts one eDNA survey unit at 0.67, and the prediction interval runs 0.13 to 0.96. ============================================================================== PART 3. SMALL-STUDY BIAS: FUNNEL AND EGGER ============================================================================== Egger regression of the standard normal deviate on precision intercept -0.940 (SE 0.902) t -1.042 on 10 df p 0.3218 slope +1.366 Funnel coordinates (x = log OR, y = standard error): study log OR SE Quilumbaquin 2023 +1.263 0.121 Eiler 2018 -0.084 0.414 Schmelzle 2016 +1.493 0.569 Moss 2022 -0.181 0.619 Valentini 2016 +3.153 0.627 Li 2024 +1.313 0.673 Di Girolamo 2024 -0.523 0.815 Biggs 2015 +3.443 0.843 Akre 2019 -1.792 0.977 Rose 2019 -1.287 1.057 Hinlo 2017 +2.178 1.563 Dougherty 2025 -2.457 1.625 Pseudo 95% funnel at SE = 1.63 spans -2.468 to +3.903 around the pooled mean ============================================================================== PART 4. LEAVE-ONE-OUT SENSITIVITY ============================================================================== omitted pooled 95% CI I2 (%) tau2 ------------------------------------------------------------------------------ Biggs 2015 +0.493 [-0.277, +1.263] 79.5 1.0883 Valentini 2016 +0.471 [-0.298, +1.240] 78.1 1.0540 Schmelzle 2016 +0.618 [-0.233, +1.469] 82.1 1.3945 Hinlo 2017 +0.650 [-0.140, +1.441] 82.1 1.2365 Eiler 2018 +0.809 [-0.024, +1.642] 78.8 1.2949 Rose 2019 +0.862 [+0.088, +1.637] 80.4 1.1299 Akre 2019 +0.917 [+0.165, +1.670] 78.8 1.0329 Moss 2022 +0.810 [-0.004, +1.624] 80.7 1.2397 Quilumbaquin 2023 +0.596 [-0.422, +1.613] 80.5 2.1924 Li 2024 +0.646 [-0.191, +1.484] 82.2 1.3492 Di Girolamo 2024 +0.829 [+0.032, +1.625] 80.8 1.1915 Dougherty 2025 +0.851 [+0.091, +1.612] 80.6 1.1155 ------------------------------------------------------------------------------ range of the pooled estimate under leave-one-out: +0.471 to +0.917 as odds ratios: 1.60 to 2.50 sign changes: 0 of 12; intervals containing zero: 8 of 12 ============================================================================== PART 5. SUBGROUPS AND MODERATORS ============================================================================== (a) By survey unit. A 'sample' contrast compares one bottle of water with one net haul. A 'visit' or 'site-season' contrast compares whole survey rounds, which usually bundle several water samples against a single conventional effort. round k = 9 pooled +1.023 [+0.167, +1.879] OR 2.78 I2 80.7% sample k = 3 pooled -0.409 [-2.705, +1.887] OR 0.66 I2 82.2% difference (sample minus round) -1.432 (SE 1.250), z = -1.15, p = 0.252 (b) By broad taxon. amphibian k = 6 pooled +1.369 [+0.441, +2.297] I2 84.2% fish k = 3 pooled +0.637 [-1.710, +2.984] I2 65.0% mammal k = 1 single contrast -0.523 (SE 0.815) reptile k = 2 pooled -1.559 [-2.965, -0.153] I2 0.0% (c) Meta-regression on log10 of the water volume processed, restricted to the 11 contrasts that report it. slope -1.416 per decade of litres (SE 0.736), z = -1.92, p = 0.055 intercept +0.352 Volumes in the set span 0.09 L to 12.00 L, a factor of 133. (d) Within-study evidence on volume, from the one study that varied it experimentally (Lopes 2017, 20 L vs 60 L samples): species p(20 L) p(60 L) log OR Hylodes phyllodes 0.614 0.761 +0.694 Hylodes asper 0.570 0.649 +0.333 Cycloramphus boraceiensis 0.154 0.596 +2.092 mean log OR for tripling volume: +1.040 (odds ratio 2.83) ============================================================================== PART 6. THE FALSE POSITIVE PROBLEM ============================================================================== Everything above treats a positive as truth. It is not. Write p11 for the probability that an occupied site gives a positive (true positive rate) and p10 for the probability that an unoccupied site gives one anyway, through contamination, primer cross-reactivity, DNA transported from upstream, or a tag-jump in the sequencing run. With prior occupancy psi, one positive sample carries a posterior probability of occupancy PPV = psi*p11 / ( psi*p11 + (1-psi)*p10 ) The method's whole selling point is rare species. Rare means small psi. And PPV falls off a cliff as psi falls, at any fixed p10. With p11 = 0.60, one positive sample: psi p10=0.001 p10=0.005 p10=0.010 p10=0.020 p10=0.050 0.500 0.998 0.992 0.984 0.968 0.923 0.200 0.993 0.968 0.938 0.882 0.750 0.100 0.985 0.930 0.870 0.769 0.571 0.050 0.969 0.863 0.759 0.612 0.387 0.020 0.924 0.710 0.550 0.380 0.197 0.010 0.858 0.548 0.377 0.233 0.108 0.005 0.751 0.376 0.232 0.131 0.057 Requiring r positive replicates out of r before calling a detection, with independent replicates: r p11^r p10^r PPV at psi=0.02 1 0.60000 0.02000000 0.3797 2 0.36000 0.00040000 0.9484 3 0.21600 0.00000800 0.9982 4 0.12960 0.00000016 0.9999 5 0.07776 0.00000000 1.0000 The cost of that rule is sensitivity: p11^r falls from 0.600 to 0.078 as r goes from 1 to 5, so the false negative rate climbs from 0.40 to 0.92 per sample. What a false positive rate does to the pooled comparison. Suppose eDNA carries p10 = 0.02 and the conventional method carries none. At psi = 0.05, the apparent eDNA detection rate at a randomly chosen site is psi*p11 + (1-psi)*p10, and the share of those positives that are real is only: psi = 0.30 -> apparent positive rate 0.1940, of which 92.8% real psi = 0.10 -> apparent positive rate 0.0780, of which 76.9% real psi = 0.05 -> apparent positive rate 0.0490, of which 61.2% real psi = 0.02 -> apparent positive rate 0.0316, of which 38.0% real ============================================================================== PART 7. NUMBERS QUOTED IN THE ARTICLE ============================================================================== pooled log OR +0.718 pooled odds ratio 2.05 [0.95, 4.41] p value 0.0661 Q 56.32 on 11 df, p = 0.00000 I-squared 80.5% tau 1.096 log odds 95% prediction interval 0.15 to 27.37 as odds ratios Egger intercept -0.940, p = 0.322 leave-one-out OR range 1.60 to 2.50 studies pooled 12 studies found and excluded 7 PPV, psi=0.02, p10=0.01, r=1 0.550 PPV, psi=0.02, p10=0.02, r=2 0.948 ============================================================================== end of output ==============================================================================