Multiple statistical comparisons in systematic reviews: A quantification of the issue in reviews of anaesthesiological interventions

Article type
Authors
Imberger G1, Damgaard-Vejlby A2, Bohnstedt-Hansen S2, Møller A1, Wetterslev J3
1Cochrane Anaesthesia Review Group, Copenhagen, Denmark
2Herlev Hospital, Herlev, Denmark
3Copenhagen Trial Unit, Centre of Clinical Intervention Research, Rigshospitalet, Copenhagen, Denmark
Abstract
Background: Systematic reviews with meta-analyses often contain high numbers of statistical comparisons. Multiplicity increases the risk of type I error beyond the nominal chosen 5% and may result in spurious conclusions. Few attempts have been made to address this problem. This omission is concerning because systematic reviews are considered to be the highest quality of evidence. Because of the emphasis on bias evaluation and structure and because of the editorial processes involved, Cochrane reviews may contain more multiplicity than their non-Cochrane counterparts. Methods: We examined all the systematic reviews published by the Cochrane Anaesthesia Review Group (CARG) containing a meta-analysis. These reviews were matched according to type of intervention and years of publication with equivalent non-Cochrane reviews. Three investigators independently counted the number of pooled comparisons in each review. Results: The over-allmedian number of pooled comparisonswas 10 (IQR 6 to 19) with 12 (range: 1 to 1872) in the Cochrane reviews and 8 (range: 1 to 62) in the non-Cochrane reviews (P=0.01). Sensitivity analyses based on bias-control of included trials were in 42% of Cochrane and 28% of non-Cochrane reviews and there was a difference in the number of comparisons added due to these analyses (P=0.02). A primary outcome was quoted in 63% of Cochrane and 51% of non-Cochrane reviews. The issue of multiple comparisons was addressed in 2% of Cochrane and 7% of non-Cochrane reviews. Conclusion: The quantity of multiplicity varies substantially but is high in systematic reviews of anaesthesiological interventions. CARG Cochrane reviews contain a statistically significant higher number of pooled comparisons compared with their non-Cochrane counterparts, possibly due to more analyses of the impact of bias control on the pooled estimates. It is concerning that few efforts are made to address sources of increased type-I error risk in systematic reviews.