Article type
Year
Abstract
Background:
A random-effects model is typically used to dealing with unexplained heterogeneity in meta-analysis. However, when the number of studies available for analysis is small, accuracy of asymptotic results may be compromised.
Objectives:
To compare alternative statistical methods for testing intervention effect with random-effects model.
Methods:
Using data extracted from The Cochrane Library, empirically we compared statistical significance levels for testing intervention effect based on: 1) currently used methods that assume large sample; 2) second-order likelihood method based on Skovgaard’s statistic; 3) bootstrap-based approach. We focused on continuous outcomes and investigated three effect measures (mean difference (MD), standardized mean difference (SMD), ratio of means (RoM)), and three methods of estimation of the heterogeneity parameter (DerSimonian-Laird (DL), maximum likelihood (ML), and restricted maximum likelihood (REML)).
Results:
After excluding meta-analyses for which the heterogeneity parameter is significantly different from zero, 66 meta-analyses were used with MD and 106 meta-analyses were analyzed with SMD and RoM. For a large majority of the analyses, P values were smaller for the traditional methods. With MD as effect measure and heterogeneity estimated by ML, we observed a significant discrepancy in P values between Skovgaard’s and the traditional Z test (mean difference = 0.05, SD of difference = 0.09). Overall, similar patterns emerged when we used the other effect measures (SMD, RoM) as well as alternative methods of estimating the heterogeneity parameter (DL, REML). The bootstrap-based significance levels were generally closer to the higher-order methods rather than the Z test.
Conclusions:
Inferential methods in random-effects model need to be carefully assessed when the number of studies is small. Alternative statistical procedures are available and easy to compute.
A random-effects model is typically used to dealing with unexplained heterogeneity in meta-analysis. However, when the number of studies available for analysis is small, accuracy of asymptotic results may be compromised.
Objectives:
To compare alternative statistical methods for testing intervention effect with random-effects model.
Methods:
Using data extracted from The Cochrane Library, empirically we compared statistical significance levels for testing intervention effect based on: 1) currently used methods that assume large sample; 2) second-order likelihood method based on Skovgaard’s statistic; 3) bootstrap-based approach. We focused on continuous outcomes and investigated three effect measures (mean difference (MD), standardized mean difference (SMD), ratio of means (RoM)), and three methods of estimation of the heterogeneity parameter (DerSimonian-Laird (DL), maximum likelihood (ML), and restricted maximum likelihood (REML)).
Results:
After excluding meta-analyses for which the heterogeneity parameter is significantly different from zero, 66 meta-analyses were used with MD and 106 meta-analyses were analyzed with SMD and RoM. For a large majority of the analyses, P values were smaller for the traditional methods. With MD as effect measure and heterogeneity estimated by ML, we observed a significant discrepancy in P values between Skovgaard’s and the traditional Z test (mean difference = 0.05, SD of difference = 0.09). Overall, similar patterns emerged when we used the other effect measures (SMD, RoM) as well as alternative methods of estimating the heterogeneity parameter (DL, REML). The bootstrap-based significance levels were generally closer to the higher-order methods rather than the Z test.
Conclusions:
Inferential methods in random-effects model need to be carefully assessed when the number of studies is small. Alternative statistical procedures are available and easy to compute.