Article type
Year
Abstract
Background: Critics have suggested that the widely used DerSimonian and Laird (DL) method for summarizing random effects often has inappropriately narrow confidence intervals and high type I error rates. The Hartung-Knapp-Sidik-Jonkman (HKSJ) method represents a popular alternative with allegedly superior properties.
Objectives: To illustrate the advisability of scepticism about unquestioning reliance on any one random-effects model.
Methods: We conducted a systematic review and meta-analysis addressing the desirability of adjunctive administration of corticosteroids in patients with community-acquired pneumonia. We tested four a priori subgroup hypotheses, including the severity of pneumonia (expecting a larger reduction in mechanical ventilation in patients trials with > 70% of patients with severe pneumonia).
Results: Random-effects meta-analysis with HKSJ and DL approaches provided identical point estimates and very similar confidence intervals, suggesting that steroids apparently reduce the need for mechanical ventilation: relative risk 0.45, 95% confidence intervals: HKSJ 0.26 to 0.79, DL 0.25 to 0.82 (Fig). For the severe pneumonia subgroup, the HKSJ confidence interval was unrealistically narrow (0.50 to 0.58) and much narrower than the intuitively sensible DL confidence interval (0.28 to 1.04). For the less severe subgroup, the HKSJ confidence interval (0.08 to 0.43) was also narrower than the DL estimate (0.04 to 0.78). This led to a statistically significant interaction P value with the HKSJ but not the DL approach: P = 0.01 and 0.18, respectively.
Conclusions: The HKSJ method for calculating confidence intervals in random-effects meta-analysis can lead to implausibly narrow confidence intervals and, in this case, suggested a spurious subgroup finding. We recommend that systematic reviewers remain alert to counterintuitive implausible statistical analysis results and, when observed, use alternative approaches.
Objectives: To illustrate the advisability of scepticism about unquestioning reliance on any one random-effects model.
Methods: We conducted a systematic review and meta-analysis addressing the desirability of adjunctive administration of corticosteroids in patients with community-acquired pneumonia. We tested four a priori subgroup hypotheses, including the severity of pneumonia (expecting a larger reduction in mechanical ventilation in patients trials with > 70% of patients with severe pneumonia).
Results: Random-effects meta-analysis with HKSJ and DL approaches provided identical point estimates and very similar confidence intervals, suggesting that steroids apparently reduce the need for mechanical ventilation: relative risk 0.45, 95% confidence intervals: HKSJ 0.26 to 0.79, DL 0.25 to 0.82 (Fig). For the severe pneumonia subgroup, the HKSJ confidence interval was unrealistically narrow (0.50 to 0.58) and much narrower than the intuitively sensible DL confidence interval (0.28 to 1.04). For the less severe subgroup, the HKSJ confidence interval (0.08 to 0.43) was also narrower than the DL estimate (0.04 to 0.78). This led to a statistically significant interaction P value with the HKSJ but not the DL approach: P = 0.01 and 0.18, respectively.
Conclusions: The HKSJ method for calculating confidence intervals in random-effects meta-analysis can lead to implausibly narrow confidence intervals and, in this case, suggested a spurious subgroup finding. We recommend that systematic reviewers remain alert to counterintuitive implausible statistical analysis results and, when observed, use alternative approaches.