Article type
Year
Abstract
Background: The majority of meta-analyses are under-powered. The risk of type 1 error in early meta-analyses is higher than the conventional 95% confidence interval (CI) and the associated probability of 5% suggests. Several techniques have been used with the goal of accounting for increased random type I error in the context of sparse data and repeated updates in meta-analysis. This study aimed to explore the role of one of these techniques – trial sequential analysis (TSA) – in assessing the reliability of conclusions in early meta-analyses.
Methods: We screened the Cochrane Database of Systematic Reviews and selected 100 meta-analyses that were large enough to have demonstrated, to a reasonable level, that the given intervention does not cause a clinically relevant effect on the outcome in question. We conducted retrospective cumulative meta-analysis using conventional techniques and measured the proportion of false positives that would have occurred. For these false positives, we performed TSA, mimicking how a prospective analysis could have been performed had it been done at the time of publication of each new trial. We used three different TSA approaches to mimic this prospective analysis. As a post-hoc analysis, we surveyed three years of Cochrane Systematic Reviews and calculated the relative risk of a meta-analysis being updated if it was not significant relative to if it were significant.
Results: Using conventional retrospective cumulative meta-analysis, one or more false positives were present in seven of the meta-analyses (7%; 95% CI 3% to 14%). Using the three TSA approaches, TSA prevented the false positive type 1 error in 13 of the 14 times the conventional threshold was crossed (93%, 95% CI 64% to 100%). Having a non-significant result made a meta-analysis from a Cochrane Systematic Review in the years 2005 to 2007 1.57 times more likely to be updated (95% CI 0.92 to 2.68).
Conclusion: TSA is a helpful statistical methodology when assessing the reliability of early nominally statistically significant findings in cumulative meta-analyses.
Methods: We screened the Cochrane Database of Systematic Reviews and selected 100 meta-analyses that were large enough to have demonstrated, to a reasonable level, that the given intervention does not cause a clinically relevant effect on the outcome in question. We conducted retrospective cumulative meta-analysis using conventional techniques and measured the proportion of false positives that would have occurred. For these false positives, we performed TSA, mimicking how a prospective analysis could have been performed had it been done at the time of publication of each new trial. We used three different TSA approaches to mimic this prospective analysis. As a post-hoc analysis, we surveyed three years of Cochrane Systematic Reviews and calculated the relative risk of a meta-analysis being updated if it was not significant relative to if it were significant.
Results: Using conventional retrospective cumulative meta-analysis, one or more false positives were present in seven of the meta-analyses (7%; 95% CI 3% to 14%). Using the three TSA approaches, TSA prevented the false positive type 1 error in 13 of the 14 times the conventional threshold was crossed (93%, 95% CI 64% to 100%). Having a non-significant result made a meta-analysis from a Cochrane Systematic Review in the years 2005 to 2007 1.57 times more likely to be updated (95% CI 0.92 to 2.68).
Conclusion: TSA is a helpful statistical methodology when assessing the reliability of early nominally statistically significant findings in cumulative meta-analyses.