Do we need comprehensive literature searches? A study Of publication and language bias in meta-analyses of controlled trials.

Tags: Oral
Bartlett C, Egger M, Jüni P, Sterne J

Background: Comprehensive literature searches are widely recommended to minimise bias in meta-analysis. Such searches are, however, time-consuming and costly. We studied the impact of unpublished and non-English language trials on effect estimates in a sample of meta-analyses.

Methods: We searched for meta-analyses in journals and the Cochrane Database of Systematic Reviews that combined (5 trials, had binary outcomes and employed comprehensive literature searches. We identified 159 meta-analyses, 60 of which included trials from grey literature (such as proceedings, theses, books, unpublished data) and 50 of which included trials published in non-English languages. We used the analytical method of the original meta-analysis to assess the impact of excluding unpublished or non-English language trials.

Results: Where present, grey literature contributed an average of 22% of the trials and 18% of the weight. Treatment effect estimates in unpublished trials were on average 7% (95% CI -2% to 18%) less beneficial than in the published trials. We calculated the change in combined treatment effect estimates when grey trials were excluded. In 41 (68%) meta-analyses the changes were <5%. In the 19 meta-analyses in which the change in estimate was (5%, 9 showed increased and 8 showed decreased benefit. Among published trials those published in non-English language journals contributed an average 21% of the trials and 17% of the weight. Treatment effect estimates were on average 16% (3% to 26%) more beneficial in non-English trials. In 29 meta-analyses (58%) the changes were <5% when excluding non-English trials. In the 21 in which the change in effect estimate was (5% 5 showed increased and 16 showed decreased benefit.

Conclusions: Exclusion of grey literature tended to increase estimated treatment benefits but the variability between meta-analyses means that the effect of ignoring grey literature is unpredictable. In general, exclusion of non-English literature decreased estimated treatment benefits, although again there was substantial between-meta-analysis heterogeneity. This is in contrast to previous studies which suggested that in conventional medicine exclusion of non-English language trials tends to increase estimated treatment benefits.