Article type
Year
Abstract
Background: It is unlikely that all studies in a meta-analysis are estimating the same underlying effect or relationship, and the variation in these quantities is frequently referred to as (statistical) heterogeneity. Such variation can be ignored, investigated or incorporated into a random-effects meta-analysis. Most meta-analyses in the Cochrane Database of Systematic Reviews (CDSR) contain very few studies, with the consequence that the amount of heterogeneity is very poorly estimated. A natural way to proceed is to draw on empirical evidence about the likely amount of heterogeneity, for example through a Bayesian meta-analysis. However, little is known about how much heterogeneity is typically observed, or how heterogeneity varies across different outcome measures, interventions or areas of health. Objectives: To formulate empirical distributions for the amount of heterogeneity in meta-analytic data sets in Cochrane reviews, both overall and broken down by different characteristics of the meta-analyses, and to discuss the application of these distributions in new meta-analyses. Methods: With the approval of the Cochrane Collaboration Steering Group and the generous collaboration of the Information Management System team, we have analysed the complete data from Issue 1 of 2008 of CDSR. We classified each data set within the ‘Data and Analyses’ hierarchy according to types of interventions (for example, pharmaceutical, surgical etc), type of outcome (for example, mortality, mental health etc) and type of participants (by branch of medicine/health care). We analysed the data using hierarchical models, accounting for uncertainty in each variance estimate, and exploring the impact of different characteristics on the typical amount of heterogeneity as well as the extent to which it varies across meta-analyses. Results: We have completed a pilot study to ensure that the research can be completed successfully and on time. Using a simple classification scheme in 49 reviews, we observed average heterogeneity variances for log odds ratios of 0.09 among 95 objective outcomes, 0.14 among 275 semi-objective outcomes and 0.12 among 29 subjective outcomes. Although there were no statistically significant differences among these, the main study involves more than 50 times as many data. Classifications of meta-analyses for the main study are ongoing at the time of writing. Conclusions: Heterogeneity may vary according to whether the outcome is hard or soft, whether the intervention is specific or complex, and whether participants are homogenous or highly variable. Our analyses allow these differences to be quantified, and facilitate the incorporation of relevant prior information into meta-analyses of small numbers of studies. Our research has also highlighted a number of errors or problems in the reviews, which are being fed back to the Review Groups.