Statistical heterogeneity and the choice between fixed- and random-effect models: an exploratory analysis of Cochrane reviews

Article type
Authors
Ospina M1, Bond K2, Vandermeer B2
1Health Technology Assessment Unit, Institute of Health Economics, Edmonton, Alberta, Canada
2Alberta Research Centre for Health Evidence, University of Alberta, Edmonton, Alberta, Canada
Abstract
Background: Two methods of combining intervention effect estimates are used in Cochrane meta-analyses (MA): fixed-effect (assuming a single common effect underlies every study) and random-effect (assuming intervention effects follow a distribution across studies) models. There is debate about which model should be used in presence of statistical heterogeneity and to what extent Cochrane review authors ought to describe reasons for their choice of model. Objectives: 1) To describe the frequency of the use of fixed versus random effects models in Cochrane reviews, 2) to explore consistency between decisions to use fixed or random effects models and the assessment of statistical heterogeneity. Methods: A random sample of 500 reviews was taken from a database of 2,500 Cochrane reviews. Reviews that included at least one MA were selected and the following data extracted from the firstMAreported: number of trials, outcomemeasure, effects estimation model, criteria for selection of effects estimation model, heterogeneity measures, criteria for interpretation of heterogeneity. Results: Of 100 reviews (median year of first publication: 2001, interquartile range [IQR]= 1998.5, 2007; median year of last update: 2005, IQR= 2003, 2009) 63% conducted at least one MA: 44 (69.8%) used fixed-effect and 19 (30.2%) used random-effect models. Twenty-one (33%) described criteria for statistical heterogeneity. Twenty-four reviews reported the intent to pool results using a fixed-effect model in absence of heterogeneity, but to use a random-effect model otherwise. Eleven MA used fixed-effect models in presence of considerable heterogeneity (p>0.01, I2 > 50%). Additional results are presented. Conclusions: Careless use of fixed-effect or random-effect approaches toMAcan lead to misleading inferences about treatment effects. Careful consideration of methods for statistically pooling comparable but heterogeneous studies remains critical to making legitimate inferences. Explicit guidance in the Cochrane Handbook regarding the choice of statistical model is required.