Background: Heterogeneity between study characteristics or results can be a problem in any systematic review or meta-analysis of clinical trials. Identifying its presence, investigating its cause, and incorporating it into analyses all involve difficult decisions for the researcher.

Objectives: We aimed to identify problems for Cochrane reviewers regarding heterogeneity, in order to inform guidelines and needs for further research.

Methods: We selected one systematic review from each Collaborative Review Group from Issue 2, 1999 of the Cochrane Database of Systematic Reviews, and examined the review and its previously published protocol. Information was extracted on plans for dealing with heterogeneity, the presence and nature of heterogeneity between the included studies, and how the heterogeneity was addressed in the review.

Results: Our principal findings were (i) the difficulty of pre-specifying important effect modifiers for subgroup analysis or meta-regression (ii) the small numbers of studies identified in many systematic reviews, thus preventing reasonable investigations of heterogeneity, and (iii) the unresolved debate concerning fixed versus random effects meta-analyses, with a majority of studies opting for fixed effect analyses, sometimes irrespective of demonstrable heterogeneity.

Conclusion: Guidelines that address practical issues are required so that reviewers' investigations of heterogeneity do not yield spurious findings. This may involve discouraging statistical investigations such as subgroup analyses and meta-regression, rather than adopting a 'cautious' approach to their interpretation, unless a large number of studies is available.