Article type
Year
Abstract
Background: random-effects meta-analysis with the DerSimonian-Laird heterogeneity estimator has become standard in systematic reviews and statistical analysis packages, including Review Manager, but it produces high type I error rates and inappropriately narrow confidence intervals, particularly in the presence of heterogeneity. A promising alternative is the Hartung-Knapp-Sidik-Jonkman (HKSJ) approach, which is increasingly being incorporated in statistical analysis packages, and is even encouraged by some journals. However, authors have reported instances in which the HKSJ method produces counterintuitive results.
Objectives: to summarize the evidence regarding the performance of the HKSJ method, in comparison to other frequentist methods, and to provide guidance regarding its use.
Methods: we searched MEDLINE, Embase, and Web of Science for simulation and empirical studies comparing the HKSJ method with other frequentist approaches to meta-analysis (e.g. fixed-effect, random-effects, profile likelihood). We present an overview of the performance of the HKSJ method compared to other frequentist methods in various scenarios (e.g. number of studies; magnitude of heterogeneity).
Results: we screened 1164 references and found 28 relevant studies, of which 24 reported on simulations, one reported on empirical data, and three reported on both. The HKSJ method typically produced more adequate type I error rates compared to other methods, and was not influenced by the number of studies, magnitude of heterogeneity, or choice of heterogeneity estimator. However, in meta-analyses of rare events (≈ 5%), type I error rates were sometimes too high, and, when heterogeneity was very low, confidence intervals were sometimes counterintuitively narrower than other frequentist methods. Further, the HKSJ method had very low power in simulations of common meta-analytic characteristics (e.g. < 20 studies, I2 statistic > 20%).
Conclusions: although HKSJ typically outperforms other methods, we recommend that systematic review authors practice caution in its application because it occasionally produces counterintuitive results. Moreover, review authors should consider the balance between power and type I error when choosing between the HKSJ method and other frequentist approaches.
Patient or healthcare consumer involvement: patients and consumers were not involved in the design, conduct, or interpretation of this study. Our study provides guidance on statistical methods, which may improve the validity of meta-analyses, and subsequently improve patient care.
Objectives: to summarize the evidence regarding the performance of the HKSJ method, in comparison to other frequentist methods, and to provide guidance regarding its use.
Methods: we searched MEDLINE, Embase, and Web of Science for simulation and empirical studies comparing the HKSJ method with other frequentist approaches to meta-analysis (e.g. fixed-effect, random-effects, profile likelihood). We present an overview of the performance of the HKSJ method compared to other frequentist methods in various scenarios (e.g. number of studies; magnitude of heterogeneity).
Results: we screened 1164 references and found 28 relevant studies, of which 24 reported on simulations, one reported on empirical data, and three reported on both. The HKSJ method typically produced more adequate type I error rates compared to other methods, and was not influenced by the number of studies, magnitude of heterogeneity, or choice of heterogeneity estimator. However, in meta-analyses of rare events (≈ 5%), type I error rates were sometimes too high, and, when heterogeneity was very low, confidence intervals were sometimes counterintuitively narrower than other frequentist methods. Further, the HKSJ method had very low power in simulations of common meta-analytic characteristics (e.g. < 20 studies, I2 statistic > 20%).
Conclusions: although HKSJ typically outperforms other methods, we recommend that systematic review authors practice caution in its application because it occasionally produces counterintuitive results. Moreover, review authors should consider the balance between power and type I error when choosing between the HKSJ method and other frequentist approaches.
Patient or healthcare consumer involvement: patients and consumers were not involved in the design, conduct, or interpretation of this study. Our study provides guidance on statistical methods, which may improve the validity of meta-analyses, and subsequently improve patient care.