Article type
Year
Abstract
Background: For a given condition, umbrella reviews summarise the results of existing Cochrane reviews for which there are multiple competing treatments. The results are not re-analysed, but are presented as a list of pairwise results from each component meta-analysis. This lack of formal statistical analysis makes it difficult to form a coherent judgement regarding which treatment should be used, an explicit objective of the umbrella review1. Methods for simultaneously analysing multiple treatments are known as Mixed Treatment Comparisons (MTC). These can be applied to form a unified analysis, and assess consistency of the evidence base.
Objectives: To highlight the difficulties in interpreting the current umbrella format and to propose a simple procedure for MTC in umbrella reviews based on summary estimates of pairwise comparisons. We also introduce approximate methods for assessing evidence consistency in umbrella reviews.
Methods: We re-analysed data from an example umbrella review of nocturnal enuresis in children1. Using the review summary table to derive the log treatment effects and variances from each pairwise meta-analysis, we combined the summary log risk ratios under a fixed effect MTC model. Model fit was assessed using a global goodness-of-fit statistic. To establish the potential for inconsistency across treatment comparisons a network diagram was assembled. We used a generalisation of Bucher's2 test to assess the discrepancy between the direct and indirect evidence.
Results: There are 8 discrete treatments and 10 pairwise comparisons reported in the umbrella review. These are summarised as 'percentage change in events', which may encourage the reader to implicitly 'rank' the treatments. The MTC meta-analysis represents an internally coherent analysis of all the data and provides a full set of risk ratios for the 28 possible pairwise comparisons. The method also enables us to say which of the 8 treatments is 'best'. However, the goodness-of-fit measure does not support the assumption of evidence consistency and further investigation is necessary. The network-of-evidence diagram reveals three potential sources of inconsistency and our approximate test of the null hypothesis (of consistency) gave a p-value of 0.0003. The enuresis data shows major inconsistencies, the source of which should be explored further.
Conclusions: In their current form, umbrella reviews could cause confusion for the decision-maker. The underlying assumption of the umbrella review is that all the evidence can be put together to decide which treatment is 'best'1. Along with the summaries from Cochrane reviews, MTC methods should be used to provide a single coherent analysis of all treatment comparisons and to check for evidence consistency.
References
1. Russell, K. & Kiddoo, D. The Cochrane Library and nocturnal enuresis: an umbrella review. Evidence-based child health: a Cochrane review journal. 2006;1:5-8
2. Bucher HC, Guyatt GH, Griffith LE, Walter SD. The results of direct and indirect treatment comparisons in meta-analysis of randomized controlled trials. J Clin. Epidemiol. 1997;50:683-91.
Objectives: To highlight the difficulties in interpreting the current umbrella format and to propose a simple procedure for MTC in umbrella reviews based on summary estimates of pairwise comparisons. We also introduce approximate methods for assessing evidence consistency in umbrella reviews.
Methods: We re-analysed data from an example umbrella review of nocturnal enuresis in children1. Using the review summary table to derive the log treatment effects and variances from each pairwise meta-analysis, we combined the summary log risk ratios under a fixed effect MTC model. Model fit was assessed using a global goodness-of-fit statistic. To establish the potential for inconsistency across treatment comparisons a network diagram was assembled. We used a generalisation of Bucher's2 test to assess the discrepancy between the direct and indirect evidence.
Results: There are 8 discrete treatments and 10 pairwise comparisons reported in the umbrella review. These are summarised as 'percentage change in events', which may encourage the reader to implicitly 'rank' the treatments. The MTC meta-analysis represents an internally coherent analysis of all the data and provides a full set of risk ratios for the 28 possible pairwise comparisons. The method also enables us to say which of the 8 treatments is 'best'. However, the goodness-of-fit measure does not support the assumption of evidence consistency and further investigation is necessary. The network-of-evidence diagram reveals three potential sources of inconsistency and our approximate test of the null hypothesis (of consistency) gave a p-value of 0.0003. The enuresis data shows major inconsistencies, the source of which should be explored further.
Conclusions: In their current form, umbrella reviews could cause confusion for the decision-maker. The underlying assumption of the umbrella review is that all the evidence can be put together to decide which treatment is 'best'1. Along with the summaries from Cochrane reviews, MTC methods should be used to provide a single coherent analysis of all treatment comparisons and to check for evidence consistency.
References
1. Russell, K. & Kiddoo, D. The Cochrane Library and nocturnal enuresis: an umbrella review. Evidence-based child health: a Cochrane review journal. 2006;1:5-8
2. Bucher HC, Guyatt GH, Griffith LE, Walter SD. The results of direct and indirect treatment comparisons in meta-analysis of randomized controlled trials. J Clin. Epidemiol. 1997;50:683-91.