Meta-analysis has become an important tool to support decisionmaking. Different meta-analysis models have been used to address heterogeneity between studies; which may provide results with varying precision and subsequently different certainty in evidence. Thus, affecting decisions made about the care of patients with chronic medical conditions. These models have been evaluated in simulation studies, not empirically.
We aimed to empirically evaluate the precision of five meta-analysis methods. Fixed-effects methods included the inverse variance method (IV) and the t distribution (IVT). Random-effects methods included those by DerSimonian and Laird (DL), Hartung-Knapp-Sidik-Jonkman (HKSJ) and the profile likelihood (PL).
We identified meta-analyses about treatments of chronic medical conditions published in 2007-2019 in the 10 medical journals with the highest impact factor. We included meta-analyses with at least 5 randomized controlled trials and chose one binary outcome deemed to be most important to patients (e.g., mortality, stroke, and myocardial infarction). Each meta-analysis was performed using the five methods. We defined discordance between methods when either boundary of 95% confidence interval of the relative risk reduction changed by more than 0.15 (an arbitrary threshold of clinical importance). We also evaluated changes of statistical significance (two-tailed p value