An investigation of the type I error rate when testing for subgroup differences in the context of random-effects meta-analyses

Article type
Authors
Guddat C1
1IQWiG, Germany
Abstract
Background: There are different approaches to test for differences between two or more subgroups of studies in the context of a meta-analysis. The Cochrane Handbook for Systematic Reviews of Interventions refers to two methods. One is a standard test for heterogeneity across subgroup results rather than across individual study results. The second is to use meta-regression analyses.
Objectives: Our aim was to compare the performance of these two approaches with respect to the type I error rate when 10 or fewer studies are available.
Methods: Assuming the random-effects model, we have conducted a simulation study for the planed comparison. Two versions of the test for heterogeneity have been considered: one with separate estimates of the between-study variance in each subgroup, and one with a pooled estimate of this variance among all subgroups. The meta-regression was conducted using two modifications for the variance estimator of the regression coefficients by Knapp and Hartung (2003). Besides the number of studies, we varied, amongst other parameters, the extent of heterogeneity between the studies, the number of subgroups and the distribution of the studies to the subgroups.
Results: Both versions of the test for heterogeneity give extremely high error rates when the heterogeneity between the studies is large and the distribution of the studies to the subgroups is unequal. In contrast, the error rates of the F-test using either of the two considered variance estimators in the context of a meta-regression are acceptable, irrespective of the chosen parameters.
Conclusions: Overall, the F-test using the refined variance estimator by Knapp and Hartung (2003; 1. modification) is the most appropriate choice out of the evaluated tests with respect to the type I error rate when the number of studies is 10 or fewer.

References:
Knapp G, Hartung J. (2003). Improved tests for a random effects meta-regression with a single covariate. Statistics in Medicine 22(17): 2693-710.