Article type
Year
Abstract
Background: The DerSimonian and Laird (DL) random-effects model is standard in meta-analysis software packages. The model is based on an easy-to-calculate estimator for the between-study variance, which is not statistically optimal. Paule and Mandel (PM) have proposed a simple iterative algorithm for obtaining the between-study variance. Under normality assumptions Ruhkin et al showed that this is equal to the restricted maximum likelihood estimator, which is statistically optimal.
Objectives: To examine if the choice of between-study variance estimator impacts the statistical inferences in random-effects meta-analyses.
Material and methods: We took the first meta-analysis in 300 randomly selected systematic reviews from the Cochrane Library, Issue 1, 2007 that included at least 3 trials, had I2>25%, and a control group event rate larger than 10%. 68 meta-analyses were eligible. We obtained the DL and PM estimates of the between-study variance, treatment effect, and the p-value. From each meta-analysis we calculated the ratios between PM and DL between-study variance estimates and treatment effect estimates. The median and 95% inter quartile range (IQR) were obtained for the ratios. We also determined how often the resulting p-values disagreed on statistical significance using P=0.05 and P=0.01 as thresholds.
Results: The ratios of between-study variance estimates had median 1.35 and IQR (0.63-17.54). The ratios between treatment effect estimates had median 1 and IQR (0.88-1.17). In 3 meta-analyses (4%, 95% CI 1-12%) the methods disagreed using P=0.05. In 6 meta-analyses (9%, 95% CI 3-18%) the methods disagreed using P=0.01.
Discussion: The PM and DL treatment effect estimates are likely to agree as the ratio is 1 with a narrow IQR. The DL estimate of the between-study variance random-effects model is likely to disagree with the statistically optimal estimate. The latter may impact both confidence intervals and P-values. It is time to introduce statistically optimal random-effects models into meta-analytic practice.
Objectives: To examine if the choice of between-study variance estimator impacts the statistical inferences in random-effects meta-analyses.
Material and methods: We took the first meta-analysis in 300 randomly selected systematic reviews from the Cochrane Library, Issue 1, 2007 that included at least 3 trials, had I2>25%, and a control group event rate larger than 10%. 68 meta-analyses were eligible. We obtained the DL and PM estimates of the between-study variance, treatment effect, and the p-value. From each meta-analysis we calculated the ratios between PM and DL between-study variance estimates and treatment effect estimates. The median and 95% inter quartile range (IQR) were obtained for the ratios. We also determined how often the resulting p-values disagreed on statistical significance using P=0.05 and P=0.01 as thresholds.
Results: The ratios of between-study variance estimates had median 1.35 and IQR (0.63-17.54). The ratios between treatment effect estimates had median 1 and IQR (0.88-1.17). In 3 meta-analyses (4%, 95% CI 1-12%) the methods disagreed using P=0.05. In 6 meta-analyses (9%, 95% CI 3-18%) the methods disagreed using P=0.01.
Discussion: The PM and DL treatment effect estimates are likely to agree as the ratio is 1 with a narrow IQR. The DL estimate of the between-study variance random-effects model is likely to disagree with the statistically optimal estimate. The latter may impact both confidence intervals and P-values. It is time to introduce statistically optimal random-effects models into meta-analytic practice.