Sensitivity of between-study heterogeneity in meta-analysis: proposed metrics and empirical evaluation

Article type
Authors
Patsopoulos N, Evangelou E, Ioannidis J
Abstract
Background: Several approaches are available for evaluating heterogeneity in meta-analysis. Sensitivity analyses are often used, but these are often implemented in various non-standardized ways. Objectives: To develop algorithms that evaluate the change in between-study heterogeneity as one or more studies are excluded sequentially or in combination from the meta-analysis calculations. To show how these algorithms can be routinely adopted in meta-analyses as standardized sensitivity analyses for heterogeneity. Methods: We developed and implemented sequential and combinatorial algorithms that evaluate the change in between-study heterogeneity as one or more studies are excluded from the calculations. The algorithms exclude studies aiming to achieve either the maximum or the minimum final I² below a desired pre-set threshold. We applied these algorithms in databases of meta-analyses of binary outcome and > = 4 studies from Cochrane Database of Systematic Reviews (Issue 4, 2005, n = 1011) and meta-analyses of genetic associations (n = 50). Two I² thresholds were used (50% and 25%). Results: Both algorithms have succeeded in achieving the pre-specified final I² thresholds. Differences in the number of excluded studies varied from 0% to 6% depending on the database and the heterogeneity threshold, while it was common to exclude different specific studies. Among meta-analyses with initial I² > 50%, in the large majority (19 [90.5%] and 208 [85.9%]) in genetic and Cochrane meta-analyses, respectively) exclusion of one or two studies sufficed to decrease I² below 50%. Similarly, among meta-analyses with initial I² > 25%, in most cases (16 [57.1%] and 382 [81.3%], respectively) exclusion of one or two studies sufficed to decrease heterogeneity even below 25%. The number of excluded studies correlated modestly with initial estimated I² (correlation coefficients 0.52 to 0.68 depending on algorithm used). Conclusions: The proposed algorithms can be routinely applied in meta-analyses as standardized sensitivity analyses for heterogeneity. Caution is needed evaluating post hoc which specific studies are responsible for the heterogeneity.