SCIENCE JOURNALING CLUB Ocean acidification and calcifiers: sorting the response literature Random-effects meta-analysis of the log response ratio ============================================================================== ============================================================================== CHECK 1 -- can the machinery recover a known effect? ============================================================================== Simulated k = 40 studies from a known random-effects model. true effect -0.3000 (-25.9%) true tau^2 0.0400 recovered -0.2933 95% CI -0.3838 to -0.2027 recovered tau^2 0.0440 true value inside its own interval: YES coverage over 500 repeat simulations: 95.2% (nominal 95%) VERDICT: PASS ============================================================================== CHECK 2 -- does random-effects collapse onto fixed-effect at tau^2 = 0? ============================================================================== Homogeneous synthetic set, k = 6. Q = 0.0093 on 5 df (Q <= df, so DL must return tau^2 = 0) tau^2 0.0000000000 random-effects +0.0995466916 (SE 0.0626836475) fixed-effect +0.0995466916 (SE 0.0626836475) |difference| 0.000e+00 (estimate), 0.000e+00 (SE) VERDICT: PASS ============================================================================== CHECK 3 -- is the subgroup test calibrated? ============================================================================== Every taxon given the SAME true effect, so every rejection is a false positive. A calibrated test rejects 5 percent of the time. G = 8 groups, n = 3 experiments each, within-study variance 0.03. within-group tau^2 fixed-effect Q_between mixed-effects Q_M ------------------------------------------------------------------------------ 0.00 5.3% 3.9% 0.05 61.8% 11.1% 0.28 97.9% 11.1% ------------------------------------------------------------------------------ 1500 simulated programmes per row. The fixed-effect Q_between decomposition is fine when within-group heterogeneity is zero and catastrophically anticonservative when it is not. At the level of within-taxon disagreement this literature actually shows, it calls a difference significant in almost every simulated programme where no difference exists. Q_M stays usable, though it is still somewhat anticonservative at n = 3 because the residual variance is itself estimated from very few studies. VERDICT: report Q_M, and treat Q_between as a warning. ============================================================================== THE EXTRACTED DATA ============================================================================== 23 effect sizes from 16 published experiments. 20 of 23 rows carry a variance computed from reported dispersion; 3 were imputed at 0.04433. study / unit lnRR percent 95% CI, percent ------------------------------------------------------------------------------ Courtney 2013 - tropical urchin -1.408 -75.5% [ -97.9, +183.6] Comeau 2018 - coralline algae -1.330 -73.6% [ -90.4, -26.9] Maier 2009 - cold-water coral -0.789 -54.6% [ -68.8, -34.0] Schoepf 2013 - Acropora millepora -0.755 -53.0% [ -68.9, -29.0] Gazeau 2007 - mussel -0.751 -52.8% [ -67.2, -32.0] Comeau 2018 - Acropora yongei -0.478 -38.0% [ -58.2, -7.9] Riebesell 2000 - coccolithophore -0.452 -36.4% [ -59.6, +0.2] Ries 2009 - molluscs -0.400 -33.0% [ -46.5, -15.9] Comeau 2009 - pteropod -0.325 -27.8% [ -32.2, -23.1] Uthicke 2012 - foraminifer -0.230 -20.5% [ -29.6, -10.2] Langer 2009 - coccolithophore -0.167 -15.4% [ -19.0, -11.6] Lischka 2011 - pteropod -0.144 -13.4% [ -23.0, -2.7] Long 2013 - Tanner crab -0.095 -9.1% [ -39.8, +37.3] Ries 2009 - temperate coral -0.062 -6.0% [ -14.9, +3.7] Chauvin 2011 - Acropora muricata -0.044 -4.3% [ -20.0, +14.6] Edmunds 2020 - Acropora hyacinthus -0.030 -3.0% [ -30.1, +34.6] Findlay 2010 - barnacle +0.033 +3.4% [ -31.6, +56.2] Gazeau 2007 - oyster +0.037 +3.8% [ -6.5, +15.2] Comeau 2018 - Pocillopora damicornis +0.220 +24.6% [ -26.6, +111.7] Ries 2009 - crustaceans +0.245 +27.8% [ +5.6, +54.6] Ries 2009 - coralline alga +0.620 +85.9% [ +23.0, +180.8] Iglesias-Rodriguez 2008 - coccolithophore +0.679 +97.3% [ +91.7, +103.0] Ries 2009 - echinoderms +0.872 +139.3% [ +38.3, +313.9] ------------------------------------------------------------------------------ Provenance ------------------------------------------------------------------------------ Riebesell 2000 - coccolithophore source: Riebesell et al. 2000, Nature 407:364 PANGAEA 728092 (raw PIC production, pmol/cell/day); our bins 250-450 uatm control (n=22, mean 0.5649, sd 0.1732) vs 650-850 uatm (n=8, mean 0.3594, sd 0.2259). Pools E. huxleyi and G. oceanica. Gazeau 2007 - mussel source: Gazeau et al. 2007, Geophys. Res. Lett. 34:L07603 Gazeau et al. 2007 Table 1, incubation-level G in mmol CaCO3 g_FW^-1 h^-1. Control = 10 lowest-pCO2 incubations (<=756 uatm), mean 0.3390 sd 0.0946; treatment = 9 incubations at 981-1461 uatm, mean 0.1600 sd 0.0786. Gazeau 2007 - oyster source: Gazeau et al. 2007, Geophys. Res. Lett. 34:L07603 Gazeau et al. 2007 Table 2. Control = 7 incubations at 698-861 uatm, mean 0.2457 sd 0.0315; treatment = 6 at 1063-1258 uatm, mean 0.2550 sd 0.0138. Iglesias-Rodriguez 2008 - coccolithophore source: Iglesias-Rodriguez et al. 2008, Science 320:336 PANGAEA 718841, PIC production pmol/cell/day. Six independent experiments, each 300 uatm vs 750 uatm (Exp6: 280 vs 600), n=3 per cell; inverse-variance combined within the paper. Comeau 2009 - pteropod source: Comeau et al. 2009, Biogeosciences 6:1877 Comeau et al. 2009 Results: 45Ca calcification 0.36 +/- 0.027 (n=10) at pH_T 8.09 vs 0.26 +/- 0.018 (n=10) at pH_T 7.78; the paper states '28% lower'. Maier 2009 - cold-water coral source: Maier et al. 2009, Biogeosciences 6:1671 Maier et al. 2009 Table 2, Skagerrak. 15-Mar: 0.046 +/- 0.010 SE (n=8) ambient vs 0.020 +/- 0.003 (n=8) at -0.30 pH. 17-Mar: 0.021 +/- 0.004 vs 0.010 +/- 0.002. Two runs combined. Langer 2009 - coccolithophore source: Langer et al. 2009, Biogeosciences 6:2637 Langer et al. 2009 Table 3, PIC production pg/cell/day, n=3. Four E. huxleyi strains, control ~400 uatm vs the level nearest 1000 uatm; strains combined. Strain lnRRs: -0.007, -0.329, -0.317, -0.439. Ries 2009 - crustaceans source: Ries, Cohen & McCorkle 2009, Geology 37:1131 PANGAEA 733947, calcification %/day, 903 vs 409 uatm. Callinectes sapidus +0.326, Homarus americanus +0.062, Penaeus plebejus +0.584; combined. Ries 2009 - molluscs source: Ries, Cohen & McCorkle 2009, Geology 37:1131 PANGAEA 733947, 903 vs 409 uatm, nine species (Argopecten, Crassostrea, Crepidula, Littorina, Mercenaria, Mya, Mytilus, Strombus, Urosalpinx); combined. Species range -4.07 to +0.96. Ries 2009 - echinoderms source: Ries, Cohen & McCorkle 2009, Geology 37:1131 PANGAEA 733947, 903 vs 409 uatm. Arbacia punctulata +1.652, Eucidaris tribuloides -0.442; combined. See the note on the archived control-block light metadata. Ries 2009 - temperate coral source: Ries, Cohen & McCorkle 2009, Geology 37:1131 PANGAEA 733947, Oculina arbuscula, 0.1962 (n=11) at 409 uatm vs 0.1843 (n=12) at 903 uatm. Ries 2009 - coralline alga source: Ries, Cohen & McCorkle 2009, Geology 37:1131 PANGAEA 733947, Neogoniolithon sp., 0.0955 (n=10) at 409 uatm vs 0.1775 (n=10) at 903 uatm. Findlay 2010 - barnacle source: Findlay et al. 2010, Estuar. Coast. Shelf Sci. 86:675 PANGAEA 737438, Semibalanus balanoides calcification mg g^-1 d^-1, tank means only: pH 8.1 -> 7.7 at both temperatures (4.06->4.35 and 4.22->4.21). No dispersion archived; variance IMPUTED. Lischka 2011 - pteropod source: Lischka et al. 2011, Biogeosciences 8:919 PANGAEA 761910, shell increment / shell diameter. 350 uatm (n=18, mean 0.7361, sd 0.0941) vs 1100 uatm (n=15, mean 0.6373, sd 0.1269), the three temperatures pooled because temperature is crossed and balanced. Chauvin 2011 - Acropora muricata source: Chauvin, Denis & Cuet 2011, Coral Reefs 30:911 PANGAEA 771294, net calcification umol/nubbin/h, 413 vs 1134 uatm, at two sites (reef flat -0.179, back reef +0.051), n=5 each; combined. Uthicke 2012 - foraminifer source: Uthicke & Fabricius 2012, Glob. Change Biol. 18:2781 PANGAEA 831207, Marginopora vertebralis, %/day. TA1 pH 8.1 (n=11) vs 7.7 (n=12); TA2 pH 8.1 (n=12) vs 7.8 (n=12); combined. Courtney 2013 - tropical urchin source: Courtney, Westfield & Ries 2013, J. Exp. Mar. Biol. Ecol. 440:169 PANGAEA 824707, Echinometra viridis, percent buoyant-mass gain computed by us per individual: 448 uatm 15.86% (n=7, sd 10.88) vs 783 uatm 3.88% (n=8, sd 13.42). Long 2013 - Tanner crab source: Long et al. 2013, PLoS ONE 8:e60959 Long et al. 2013 Results: 'Percent calcium was higher in Control crabs than in pH 7.8 or pH 7.5 crabs by 10% and 11%'. We use the pH 7.8 figure. No SE given; variance IMPUTED. Schoepf 2013 - Acropora millepora source: Schoepf et al. 2013, PLoS ONE 8:e75049 Schoepf et al. 2013: calcification 'decreased by 53%' at 741 uatm in the second experimental half. Three congeneric species in the same paper showed no decrease. Variance IMPUTED. Comeau 2018 - Acropora yongei source: Comeau et al. 2018, Glob. Change Biol. 24:4857 PANGAEA 892655, mg CaCO3 cm^-2 d^-1. Ambient 1.6955 (n=7, sd 0.4326) vs high-DIC low-pH 1.0516 (n=7, sd 0.4927). Comeau 2018 - Pocillopora damicornis source: Comeau et al. 2018, Glob. Change Biol. 24:4857 PANGAEA 892655. Ambient 0.6996 (n=5, sd 0.3930) vs high-DIC low-pH 0.8720 (n=6, sd 0.2136). Comeau 2018 - coralline algae source: Comeau et al. 2018, Glob. Change Biol. 24:4857 PANGAEA 892655. Sporolithon durum 0.0376 (n=5) -> 0.0088 (n=5); Neogoniolithon sp. 0.0128 (n=4) -> 0.0040 (n=4); combined. Edmunds 2020 - Acropora hyacinthus source: Edmunds & Burgess 2020, J. Exp. Biol. 223:jeb217000 PANGAEA 926042, whole colonies, mg/day, low-temperature arm. Large 1046.5 (n=4) -> 989.0 (n=4); small 188.2 (n=4) -> 196.0 (n=4); combined. ============================================================================== OVERALL POOL ============================================================================== all taxa k = 23 pooled lnRR -0.1398 (SE 0.1180) 95% CI -0.3711 to +0.0916 as percent -13.0% [-31.0%, +9.6%] z = -1.184, p = 0.2364 Q = 1859.98 on 22 df, p = 0 tau^2 = 0.2775, I^2 = 98.8%, H^2 = 84.54 95% prediction -71.7% to +167.2% For contrast, the fixed-effect pool is +0.2466 (SE 0.0103). The random- effects estimate is wider because tau^2 is not zero. Nothing here is a substitute for looking at the spread. ============================================================================== BY TAXON ============================================================================== taxon k lnRR percent 95% CI (percent) ------------------------------------------------------------------------------ Mollusc 3 -0.345 -29.2% [ -54.6%, +10.4%] Coralline alga 2 -0.298 -25.7% [ -89.0%, +400.4%] Coral 7 -0.259 -22.8% [ -38.1%, -3.6%] Pteropod 2 -0.242 -21.5% [ -34.2%, -6.2%] Foraminifer 1 -0.230 -20.5% [ -29.6%, -10.2%] Coccolithophore 3 +0.042 +4.2% [ -47.9%, +108.5%] Echinoderm 2 +0.058 +6.0% [ -87.6%, +802.0%] Crustacean 3 +0.133 +14.3% [ -6.7%, +39.9%] ------------------------------------------------------------------------------ Does taxon explain anything? Two tests, two answers. Q_total = 1859.98 Q_within = 1105.64 (a) NAIVE fixed-effect decomposition Q_between = 754.34 on 7 df, p = 1.319e-158 Taken at face value this says taxon matters enormously. Check 3 says this test rejects a true null 98% of the time at the level of within-taxon heterogeneity we actually have. Discard it. (b) MIXED-EFFECTS moderator test, the one that answers the question residual tau^2 = 0.2792 (within-taxon disagreement, charged to error) Q_M = 3.14 on 7 df, p = 0.8722 Once within-taxon disagreement is charged to the error term, the differences BETWEEN taxa are indistinguishable from the differences WITHIN them. Taxon explains essentially none of the heterogeneity. Within-taxon heterogeneity (is the taxon label enough on its own?) Mollusc k=3 Q= 25.50 df=2 p=2.906e-06 I^2= 92.2% tau^2=0.1383 Coralline alga k=2 Q= 12.12 df=1 p=0.0004982 I^2= 91.8% tau^2=1.7448 Coral k=7 Q= 28.41 df=6 p=7.876e-05 I^2= 78.9% tau^2=0.0614 Pteropod k=2 Q= 7.16 df=1 p=0.007466 I^2= 86.0% tau^2=0.0141 Foraminifer k=1 (single experiment, no within-taxon test) Coccolithophore k=3 Q=1026.69 df=2 p=1.141e-223 I^2= 99.8% tau^2=0.3585 Echinoderm k=2 Q= 3.17 df=1 p=0.07505 I^2= 68.4% tau^2=1.7792 Crustacean k=3 Q= 2.60 df=2 p=0.2727 I^2= 23.0% tau^2=0.0086 ============================================================================== SMALL-STUDY BIAS ============================================================================== Egger's regression test intercept +0.4029 (SE 0.1143), t = 3.524 on 21 df, p = 0.002016 The funnel is asymmetric. Small, imprecise experiments report systematically different effects from large precise ones. Egger's test is fragile here, so drop each row in turn and refit: intercept ranges -0.177 to +0.557; p ranges 3.1e-05 to 0.00372 the single most influential row is Iglesias-Rodriguez 2008 - coccolithophore (dropping it: intercept -0.177, p = 0.00372) Funnel plot coordinates (x = lnRR, y = standard error, small at top) study / unit lnRR SE ------------------------------------------------------------------------------ Iglesias-Rodriguez 2008 - coccolithophore +0.679 0.015 Langer 2009 - coccolithophore -0.167 0.022 Comeau 2009 - pteropod -0.325 0.032 Ries 2009 - temperate coral -0.062 0.050 Gazeau 2007 - oyster +0.037 0.053 Lischka 2011 - pteropod -0.144 0.060 Uthicke 2012 - foraminifer -0.230 0.062 Chauvin 2011 - Acropora muricata -0.044 0.092 Ries 2009 - crustaceans +0.245 0.097 Ries 2009 - molluscs -0.400 0.116 Edmunds 2020 - Acropora hyacinthus -0.030 0.167 Gazeau 2007 - mussel -0.751 0.186 Maier 2009 - cold-water coral -0.789 0.191 Comeau 2018 - Acropora yongei -0.478 0.202 Ries 2009 - coralline alga +0.620 0.211 Findlay 2010 - barnacle +0.033 0.211 Long 2013 - Tanner crab -0.095 0.211 Schoepf 2013 - Acropora millepora -0.755 0.211 Riebesell 2000 - coccolithophore -0.452 0.232 Comeau 2018 - Pocillopora damicornis +0.220 0.270 Ries 2009 - echinoderms +0.872 0.280 Comeau 2018 - coralline algae -1.330 0.519 Courtney 2013 - tropical urchin -1.408 1.250 ------------------------------------------------------------------------------ Pseudo 95% limits at the pooled estimate -0.140: SE 0.312 -> -0.752 to +0.473 SE 0.625 -> -1.365 to +1.085 SE 0.937 -> -1.977 to +1.698 SE 1.250 -> -2.590 to +2.310 ============================================================================== LEAVE-ONE-OUT SENSITIVITY ============================================================================== dropped lnRR percent I^2 ------------------------------------------------------------------------------ Riebesell 2000 - coccolithophore -0.1260 -11.8% 98.9% Gazeau 2007 - mussel -0.1110 -10.5% 98.9% Gazeau 2007 - oyster -0.1502 -13.9% 98.9% Iglesias-Rodriguez 2008 - coccolithophore -0.1610 -14.9% 84.7% Comeau 2009 - pteropod -0.1297 -12.2% 98.6% Maier 2009 - cold-water coral -0.1094 -10.4% 98.9% Langer 2009 - coccolithophore -0.1413 -13.2% 98.5% Ries 2009 - crustaceans -0.1598 -14.8% 98.9% Ries 2009 - molluscs -0.1266 -11.9% 98.9% Ries 2009 - echinoderms -0.1811 -16.6% 98.9% Ries 2009 - temperate coral -0.1450 -13.5% 98.8% Ries 2009 - coralline alga -0.1742 -16.0% 98.9% Findlay 2010 - barnacle -0.1477 -13.7% 98.9% Lischka 2011 - pteropod -0.1402 -13.1% 98.8% Chauvin 2011 - Acropora muricata -0.1451 -13.5% 98.9% Uthicke 2012 - foraminifer -0.1355 -12.7% 98.8% Courtney 2013 - tropical urchin -0.1301 -12.2% 98.9% Long 2013 - Tanner crab -0.1419 -13.2% 98.9% Schoepf 2013 - Acropora millepora -0.1118 -10.6% 98.9% Comeau 2018 - Acropora yongei -0.1243 -11.7% 98.9% Comeau 2018 - Pocillopora damicornis -0.1547 -14.3% 98.9% Comeau 2018 - coralline algae -0.1086 -10.3% 98.9% Edmunds 2020 - Acropora hyacinthus -0.1451 -13.5% 98.9% ------------------------------------------------------------------------------ (none dropped) -0.1398 -13.0% 98.8% Pooled estimate ranges from -0.1811 to -0.1086 (-16.6% to -10.3%) across the leave-one-out set. The sign of the pooled effect is stable. ============================================================================== NON-INDEPENDENCE CHECK ============================================================================== Three papers contribute more than one row. Dropping the multi-row papers down to a single inverse-variance-combined row each: Courtney, Westfield & Ries 2013, J. Exp. Mar. Biol. -1.408 Maier et al. 2009, Biogeosciences 6:1671 -0.789 Schoepf et al. 2013, PLoS ONE 8:e75049 -0.755 Riebesell et al. 2000, Nature 407:364 -0.452 Comeau et al. 2018, Glob. Change Biol. 24:4857 (3 ro -0.326 Comeau et al. 2009, Biogeosciences 6:1877 -0.325 Uthicke & Fabricius 2012, Glob. Change Biol. 18:2781 -0.230 Langer et al. 2009, Biogeosciences 6:2637 -0.167 Lischka et al. 2011, Biogeosciences 8:919 -0.144 Long et al. 2013, PLoS ONE 8:e60959 -0.095 Chauvin, Denis & Cuet 2011, Coral Reefs 30:911 -0.044 Edmunds & Burgess 2020, J. Exp. Biol. 223:jeb217000 -0.030 Gazeau et al. 2007, Geophys. Res. Lett. 34:L07603 (2 -0.023 Ries, Cohen & McCorkle 2009, Geology 37:1131 (5 rows -0.005 Findlay et al. 2010, Estuar. Coast. Shelf Sci. 86:67 +0.033 Iglesias-Rodriguez et al. 2008, Science 320:336 +0.679 one row per paper k = 16 pooled lnRR -0.1804 (SE 0.1379) 95% CI -0.4508 to +0.0900 as percent -16.5% [-36.3%, +9.4%] z = -1.308, p = 0.1909 Q = 1796.73 on 15 df, p = 0 tau^2 = 0.2699, I^2 = 99.2%, H^2 = 119.78 95% prediction -73.6% to +164.4% ============================================================================== WHAT THE NUMBERS SAY ============================================================================== Pooled effect on calcification and growth across all calcifying taxa: -13.0% (95% CI -31.0% to +9.6%), k = 23 Heterogeneity: I^2 = 98.8%, tau^2 = 0.2775, Q = 1860.0 on 22 df (p = 0) Prediction interval for the next experiment: -71.7% to +167.2% Does taxonomic group explain that spread? naive fixed-effect Q_between = 754.3 on 7 df, p = 1.32e-158 (miscalibrated) mixed-effects Q_M = 3.14 on 7 df, p = 0.872 (the answer: no) Read the prediction interval before the confidence interval. The confidence interval says where the AVERAGE sits. The prediction interval says what the next experiment is likely to find, and it spans both signs. Then read Q_M. The spread between taxa looks large, but it is no larger than the spread inside a single taxon, and the honest summary of this literature is that experiments on the same organisms disagree with each other about as much as experiments on different phyla do. That is a statement about the experiments, and it is the finding. ==============================================================================