Article type
Year
Abstract
Background: A random-effects model is often used to dealing with unexplained heterogeneity in meta-analysis. However, typical applications use a small number of studies suggesting that statistical inference based on first-order asymptotic theory may lead to inaccurate results.
Objectives: To empirically evaluate performance of higher-order asymptotic methods in random effects meta-analysis.
Methods: We applied higher-order asymptotic methods to a series of meta-analyses using data extracted from the Cochrane Library. We focused on continuous outcomes and compared traditional methods with a second-order likelihood method based on Skovgaard’s statistic. We have also used a signed-likelihood ratio test approach. Three effect measures mean difference (MD), standardized mean difference (SMD), Ratio of Means (RoM), and three methods of estimation for the heterogeneity parameter DerSimonian-Laird (DL), Maximum Likelihood (ML), and Restricted Maximum Likelihood (REML) were investigated. We included meta-analyses for which the heterogeneity parameter is significantly different from zero.
Results: 66 meta-analyses were used in which the effect measure was MD. The largest average discrepancy in p-values was between Skovgaard’s method and a p-value based Z test based on ML estimator mean (SD) of difference=0.05 (0.09). For SMD and ROM, 106 meta-analyses were available for analysis. Once again the largest discrepancy for SMD occurred between Skovgaard’s and Z test with ML mean (SD) of difference = 0.03 (0.04). The results for ROM were similar to that of the SMD. In one example with 14 studies, the p-values for effect of intervention using a signed log likelihood ratio test and the second order Skovgaard’s statistic were 0.2526, and 0.635, respectively, while Z test with ML and REML resulted in p-values of 0.0547 and 0.0653, respectively.
Conclusions: Traditional first-order asymptotic methods in random-effects model might not be reliable when the number of studies is small. Higher-order asymptotics methods are available and need to be assessed in parallel.
Objectives: To empirically evaluate performance of higher-order asymptotic methods in random effects meta-analysis.
Methods: We applied higher-order asymptotic methods to a series of meta-analyses using data extracted from the Cochrane Library. We focused on continuous outcomes and compared traditional methods with a second-order likelihood method based on Skovgaard’s statistic. We have also used a signed-likelihood ratio test approach. Three effect measures mean difference (MD), standardized mean difference (SMD), Ratio of Means (RoM), and three methods of estimation for the heterogeneity parameter DerSimonian-Laird (DL), Maximum Likelihood (ML), and Restricted Maximum Likelihood (REML) were investigated. We included meta-analyses for which the heterogeneity parameter is significantly different from zero.
Results: 66 meta-analyses were used in which the effect measure was MD. The largest average discrepancy in p-values was between Skovgaard’s method and a p-value based Z test based on ML estimator mean (SD) of difference=0.05 (0.09). For SMD and ROM, 106 meta-analyses were available for analysis. Once again the largest discrepancy for SMD occurred between Skovgaard’s and Z test with ML mean (SD) of difference = 0.03 (0.04). The results for ROM were similar to that of the SMD. In one example with 14 studies, the p-values for effect of intervention using a signed log likelihood ratio test and the second order Skovgaard’s statistic were 0.2526, and 0.635, respectively, while Z test with ML and REML resulted in p-values of 0.0547 and 0.0653, respectively.
Conclusions: Traditional first-order asymptotic methods in random-effects model might not be reliable when the number of studies is small. Higher-order asymptotics methods are available and need to be assessed in parallel.