Article type
Year
Abstract
Background: Evaluating the variation in the strength of the effect across studies is a key feature of meta-analyses. This variability is reflected by measures like τ2 or I2 but their clinical interpretation is not straightforward, especially not if the meta-analysis is in odds ratios or risk ratios.
Objectives: A prediction interval presents the expected range of true study effects in similar studies. We aim to show how it can help to understand the uncertainty about whether a treatment works or not.
Methods: Conclusions based on confidence intervals (CI) may not hold in all settings. We evaluated the differences in conclusions based on 95% CIs and 95% prediction intervals in statistically significant meta-analyses published in the Cochrane Database of Systematic Reviews between 2009 and 2013. If the estimated I2 was > 0 we used this I2 for the prediction interval. When the estimated I2 was 0, we imputed low levels of heterogeneity (I2 = 20%), because heterogeneity estimates are often imprecise.
Results: A statistically significant meta-analysis does not guarantee that the treatment will be effective in all settings: in 347 (72%) of the 479 statistically significant meta-analyses with I2 > 0, the prediction interval showed that the treatment could be ineffective. Imputation of an I2 of 20% in the other 441 statistically significant meta-analyses gave similar results: 329 (75%) of the treatments could be ineffective.
Conclusions: The CI is inadequate for clinical decision making because it only summarizes the average treatment effect. The prediction interval is more informative as it shows the range of possible effects in relation to the no-effect and clinical benefit thresholds. A narrow prediction interval completely on the beneficial side of a clinically relevant threshold increases confidence in an intervention. A broad prediction interval may indicate the existence of settings where the treatment has a suboptimal and possibly even harmful effect. Prediction intervals should be routinely reported to allow more informative inferences in meta-analyses.
Objectives: A prediction interval presents the expected range of true study effects in similar studies. We aim to show how it can help to understand the uncertainty about whether a treatment works or not.
Methods: Conclusions based on confidence intervals (CI) may not hold in all settings. We evaluated the differences in conclusions based on 95% CIs and 95% prediction intervals in statistically significant meta-analyses published in the Cochrane Database of Systematic Reviews between 2009 and 2013. If the estimated I2 was > 0 we used this I2 for the prediction interval. When the estimated I2 was 0, we imputed low levels of heterogeneity (I2 = 20%), because heterogeneity estimates are often imprecise.
Results: A statistically significant meta-analysis does not guarantee that the treatment will be effective in all settings: in 347 (72%) of the 479 statistically significant meta-analyses with I2 > 0, the prediction interval showed that the treatment could be ineffective. Imputation of an I2 of 20% in the other 441 statistically significant meta-analyses gave similar results: 329 (75%) of the treatments could be ineffective.
Conclusions: The CI is inadequate for clinical decision making because it only summarizes the average treatment effect. The prediction interval is more informative as it shows the range of possible effects in relation to the no-effect and clinical benefit thresholds. A narrow prediction interval completely on the beneficial side of a clinically relevant threshold increases confidence in an intervention. A broad prediction interval may indicate the existence of settings where the treatment has a suboptimal and possibly even harmful effect. Prediction intervals should be routinely reported to allow more informative inferences in meta-analyses.