Background: The regular updating of Cochrane Reviews is a demanding, ongoing process. To focus resources, reviews in which the pooled results are inconclusive or highly likely to be altered by new studies should be prioritized over those whose results are more stable, or even definitive. As such, quantitative methods to assess the strength of evidence within meta-analyses are desirable. To monitor the accumulation of evidence over time, cumulative meta-analysis can be undertaken in which the effect size estimate is updated sequentially as new results become available.
Objectives: The aim of this study is to compare alternative statistical methods for assessing whether sufficient data have been accumulated at different points within cumulative meta-analyses, to determine whether a review update should be prioritized.
Methods: Several methods to quantify the strength of evidence within a meta-analysis were applied to a database of the systematic reviews published or updated in the Cochrane Library between 2008 and 2012. Such methods – including the calculation of monitoring boundaries based on an ‘Optimal Information Size’ – have mostly been adapted from the interim analysis of clinical trials, in which the same issues of multiplicity from repeated testing and random fluctuations in effect size are important.
Results: The methods investigated in this study were appraised by comparing their application to a number of cumulative meta-analyses. Differences in the results obtained from the various methods will be demonstrated with examples, and the suitability of the methods in different situations compared.
Conclusions: We draw conclusions about the attributes and suitability of different statistical methods of assessing the strength of evidence within a meta-analysis. Furthermore, such methods can provide an objective, evidence-based approach to recommendations for systematic review updates.