Background: Publication bias has long been recognised as a problem for systematic reviews since the likelihood of identifying studies is related to the results of those studies. One way to investigate whether a review is subject to publication bias is to use 'funnel plots' to investigate the presence of asymmetry.
Objective: To assess how many systematic reviews and protocols published by the Cochrane Drugs and Alcohol Group (CDAG) searched for unpublished studies, found it and analyzed the possibility of publication bias through a funnel plot.
Methods: We analysed all reviews/protocols published by the CDAG on the Cochrane Library, Issue 2, 2007. We analyzed: 1. how many reviews and protocols searched for unpublished studies 2. how many reviews and protocols described the conference proceedings searched 3. how many of these reviews found unpublished studies or unpublished data of published studies 4. how many reviews which found unpublished studies included them in the results 5. how many reviews which included unpublished studies also performed a sensitivity analysis excluding them. 6. how many reviews and protocols analyzed the possibility of publication bias using funnel plot or others statistical methods 7. how many reviews discussed the possibility of publication bias.
Results: We analyzed 37 reviews and 15 protocols.
Reviews and protocols which contacted authors and pharmaceutical companies: 36/52 (15/15 protocols and 21/37 reviews)
Reviews and protocols which searched conference proceedings: 32/52 (11/15 protocols and 21/37 reviews)
Reviews and protocols which described the conference proceedings searched: 10/32 (4/11 protocols and 6/21 reviews)
Reviews which found unpublished studies: 4/21
Reviews which found unpublished data of published studies: 9/21
Reviews which included unpublished studies in the analysis: 4/21. Each review found one unpublished study. Three reviews performed a meta-analysis and the unpublished studies contributed data for 7 outcomes
Reviews which performed a sensitivity analysis excluding unpublished studies: 0/4
Reviews and protocols which analysed the possibility of publication bias by a funnel plot: 4/50 (two reviews didn't include any studies in the review) (3 protocols, 1 review)
Reviews which discussed the possibility of publication bias: 0/37
Conclusions: These results show that publication bias is not properly considered by review authors. 69% of the reviews and protocols declared that they would search for unpublished studies by contacting authors and pharmaceutical companies and 61% by searching conference proceedings, but only 31% of these described the conference proceedings searched. Only 19% of reviews which searched for unpublished studies found them. 43% of reviews included unpublished data of published studies. The results did not state clearly if unpublished studies had been found and included in the analysis. Only 8% of the reviews explored the possibility of publication bias by a funnel plot or other statistical methods and none of the reviews discussed possible publication bias in the discussion section of the review. Nevertheless we found an improvement over time: all the protocols stated their intention to search for unpublished studies while only the 57% of the reviews did this. More attention should be given by the editorial team of each review group to this problem by: giving each review author information about publication bias and its importance; inviting authors to search for unpublished studies, to use the funnel plot and to discuss them in the analysis, providing instructions on how to conduct the analysis; inviting authors to report, at the beginning of "description of studies" section, how many unpublished studies have been retrieved and included in the analysis.