Article type
Year
Abstract
Background: Meta-analyses may commit Type 1 errors due to inclusion of biased trials and multiple testing when updating. To account for such errors, meta-analyses may be analysed with trial sequential boundaries (TSB) to determine when convincing evidence is reached1,2. TSB require calculation of an optimal information size (OIS) (i.e. the sample size needed to detect a realistic intervention effect). OIS could be based on results from low-bias risk trials1 and could consider heterogeneity among the included trials2.
Objectives: To examine meta-analyses for potential spurious P<0.05 values with TSB based on optimal heterogeneity-based information size2.
Material: All meta-analyses reporting a binary outcome from the 189 Cochrane Neonatal Group reviews (The Cochrane Library
2004, Issue 4). We selected all meta-analyses that included more than two trials and had at least one trial with or without adequate allocation concealment. A maximum of three outcomes was meta-analysed from each review.
Methods: We defined2 optimal heterogeneity-based information size = OIS/(1-I2). OIS was calculated from the pooled intervention effect of trials with adequate allocation concealment for alpha=5% and 1-Beta=80%1. The cumulated Z-value, the traditional criterion Z=1.96 (two-sided P=0.05), and the heterogeneity based TSB were constructed.
Results: One hundred and forty-five meta-analyses were eligible. For 74 meta-analyses (51%) the cumulated Z-curves never crossed Z=1.96 and heterogeneity based TSB. Fourteen Z-curves (10%) crossed Z=1.96 temporarily but returned to non-significant values. Of these, two Z-curves also crossed the TSB temporarily. Fifty-seven Z-curves (39%) permanently crossed Z=1.96. Of these, 39 presented firm evidence for a beneficial intervention effect as the Z-curve also crossed TSB, but 18 Z-curves never crossed the TSB. The latter may represent potential spurious P-values.
Conclusions: We found 18/57 (32%; 95%-CI: 26-38%) meta-analyses with significant P<0.05 which might be spurious according to heterogeneity based TSB. The impact of these findings on the overall conclusion in the specific neonatal review should be further investigated.
References
1. Brok J, Thorlund K, Gluud C, Wetterslev J. Trial sequential analyses of six Cochrane Neonatal Group meta-analyses considering adequacy of allocation concealment (II). Clinical Trials 2005; 2(Suppl 1):S61-2.
2. Thorlund K, Wetterslev J, Brok J, Gluud C. Trial sequential analyses of six meta-analyses considering heterogeneity and trial weight (III). Clinical Trials 2005; 2(Suppl 1):S62.
Objectives: To examine meta-analyses for potential spurious P<0.05 values with TSB based on optimal heterogeneity-based information size2.
Material: All meta-analyses reporting a binary outcome from the 189 Cochrane Neonatal Group reviews (The Cochrane Library
2004, Issue 4). We selected all meta-analyses that included more than two trials and had at least one trial with or without adequate allocation concealment. A maximum of three outcomes was meta-analysed from each review.
Methods: We defined2 optimal heterogeneity-based information size = OIS/(1-I2). OIS was calculated from the pooled intervention effect of trials with adequate allocation concealment for alpha=5% and 1-Beta=80%1. The cumulated Z-value, the traditional criterion Z=1.96 (two-sided P=0.05), and the heterogeneity based TSB were constructed.
Results: One hundred and forty-five meta-analyses were eligible. For 74 meta-analyses (51%) the cumulated Z-curves never crossed Z=1.96 and heterogeneity based TSB. Fourteen Z-curves (10%) crossed Z=1.96 temporarily but returned to non-significant values. Of these, two Z-curves also crossed the TSB temporarily. Fifty-seven Z-curves (39%) permanently crossed Z=1.96. Of these, 39 presented firm evidence for a beneficial intervention effect as the Z-curve also crossed TSB, but 18 Z-curves never crossed the TSB. The latter may represent potential spurious P-values.
Conclusions: We found 18/57 (32%; 95%-CI: 26-38%) meta-analyses with significant P<0.05 which might be spurious according to heterogeneity based TSB. The impact of these findings on the overall conclusion in the specific neonatal review should be further investigated.
References
1. Brok J, Thorlund K, Gluud C, Wetterslev J. Trial sequential analyses of six Cochrane Neonatal Group meta-analyses considering adequacy of allocation concealment (II). Clinical Trials 2005; 2(Suppl 1):S61-2.
2. Thorlund K, Wetterslev J, Brok J, Gluud C. Trial sequential analyses of six meta-analyses considering heterogeneity and trial weight (III). Clinical Trials 2005; 2(Suppl 1):S62.