Article type
Year
Abstract
Introduction: Additional clinical trials may be performed or identified after the publication of a systematic review on the Cochrane Database of Systematic Reviews (CDSR). One of the principal assets of the Cochrane Library being published electronically is that reviews can be updated with information from new trials. However, statistical problems may be encountered and interpretation can be difficult. Observing whether a confidence interval crosses the line of no effect is equivalent to performing a significance test, and repeated significance tests can be misleading. What is one to conclude if treatment appears significantly better than control on the basis of four trials but not when a fifth is added to the review?
Objectives: (i) To determine the prevalence and consequences of updating meta-analyses on CDSR and (ii) to outline approaches which might be taken to aid the interpretation of updated reviews.
Methods: A survey of all Cochrane reviews up to the end of 1998 has been conducted. Reviews which had been updated with additional trials were identified. A primary outcome was determined for each such review using a pre-specified rule. Meta-analyses were performed for the selected outcome at both the first and latest appearance of the review in the CDSR. It was noted whether each result was significant at a 5% level. Changes in statistical significance were counted.
Results: Of 481 systematic reviews in the 1998 issue 4 release of the Cochrane Library, 65 had gained at least one additional trial since first appearing. Random effects meta-analyses on the primary outcome changed significance over time in five of these reviews: one from significant to not significant and four vice versa. Seven fixed effect analyses on the primary outcome changed in significance. In addition, of 708 repeated Peto odds ratio analyses among the 65 reviews, 17 changed from significance to non-significance and 45 vice versa.
Discussion: Changing results on the addition of new trials to a systematic review are not rare. I discuss implications of these findings for the Cochrane Collaboration and mention two approaches which might theoretically be taken to address statistical and interpretation problems. The first is to take a Bayesian approach to meta-analysis from the outset. The second is to use formal sequential procedures as implemented in sequential clinical trials.
Objectives: (i) To determine the prevalence and consequences of updating meta-analyses on CDSR and (ii) to outline approaches which might be taken to aid the interpretation of updated reviews.
Methods: A survey of all Cochrane reviews up to the end of 1998 has been conducted. Reviews which had been updated with additional trials were identified. A primary outcome was determined for each such review using a pre-specified rule. Meta-analyses were performed for the selected outcome at both the first and latest appearance of the review in the CDSR. It was noted whether each result was significant at a 5% level. Changes in statistical significance were counted.
Results: Of 481 systematic reviews in the 1998 issue 4 release of the Cochrane Library, 65 had gained at least one additional trial since first appearing. Random effects meta-analyses on the primary outcome changed significance over time in five of these reviews: one from significant to not significant and four vice versa. Seven fixed effect analyses on the primary outcome changed in significance. In addition, of 708 repeated Peto odds ratio analyses among the 65 reviews, 17 changed from significance to non-significance and 45 vice versa.
Discussion: Changing results on the addition of new trials to a systematic review are not rare. I discuss implications of these findings for the Cochrane Collaboration and mention two approaches which might theoretically be taken to address statistical and interpretation problems. The first is to take a Bayesian approach to meta-analysis from the outset. The second is to use formal sequential procedures as implemented in sequential clinical trials.