Background: In most network meta-analyses (NMAs), substantial variability in results makes random-effects an appealing model. A potential advantage of NMAs is the possibility that consistent results from direct and indirect comparisons will narrow confidence intervals (CIs) in comparison to those from direct estimates alone. If direct comparisons provide higher quality evidence than network estimates, clinicians should use that higher certainty evidence to guide their practice. Objective: To illustrate problems in the use of random-effect models in Bayesian NMAs. Method: We conducted a number of Bayesian NMAs where we first used random-effects, found counter-intuitive results, and then applied fixed-effects models. Results: Applying the GRADE (Grading of Recommendations Assessment, Development and Evaluation) approach, we used random-effects models to generate direct, indirect, and network estimates. We found instances where direct and indirect estimates were similar, but CIs of NMA estimates were far wider than the direct estimates. In these instances, fixed-effects models generated narrower CIs consistent with the direct estimates, e.g. in an NMA of alternative fluids for resuscitation in septic shock, random-effects showed very similar point estimates for direct and indirect comparisons; the CI around the direct estimate was far narrower (Table 1). Counter-intuitively, the NMA estimate was far wider than the direct estimate. In the fixed-effect model, the sparse data from the indirect estimate changed the NMA estimate little in comparison to the direct estimate – as one would intuitively expect (Table 1). We have encountered a number of such situations. As in this example, in the face of similar point estimates from direct and indirect comparisons, basing clinical decision-making on an NMA estimate that is far less precise than the direct estimate is inappropriate. Options are to use the direct estimate only, or apply a fixed-effects model that generates sensible results.
Conclusions: Those conducting Bayesian NMA need to be aware of potential problems with random-effect models and, when counter-intuitive results arise, consider fixed-effects models.