Article type
Year
Abstract
Background: Publication bias is widely discussed and appreciated within the Cochrane Collaboration. However, it is unclear whether reviewers are taking practical steps to avoid publication bias by trying to identify and include data from unpublished trials in their reviews. Objectives To assess whether Cochrane reviewers search for unpublished trials and whether information from such trials is included in reviews. To assess whether the possibility of publication bias is discussed within reviews.
Methods: A sample was obtained by selecting all completed reviews listed as new items in the Cochrane Library 2000, Issue 1. Reviews were scored as to whether their search strategies specifically mentioned a. any search method more extensive than electronic bibliographic searches and searches of their CRG specialist register, b. communicating directly with trialists etc. c. seeking unpublished trials. The number of trials included in each review that were a. published as full papers, b. published as abstracts c. unpublished were counted. For each review a text search using the phrase "publication bias" was done and outcome summarised.
Results: Of 52 completed systematic reviews listed as new items, 71% reported search strategies more extensive than bibliographic searching, 75% reported communication with trialists etc. and 52% specifically mentioned seeking unpublished trials. Of the 51 systematic reviews that identified eligible trials, 59% included only published trials, 29% included trials reported as abstracts or letters and 20% included unpublished trials. Of the 10 reviews that included unpublished trials, half reported a search strategy seeking such trials while the other half made no mention of looking for unpublished trials in their search strategy. 12 reviews used the phrase publication bias of which 10 planned to use funnel plots to assess the possibility of publication bias. Owing to inadequate or inappropriate data, only 4 were able to do so.
Discussion: Only 52% of the sample of Cochrane systematic reviews reported actively searching for unpublished trials, although 75% reported asking a variety of people to help identify trials, which may implicitly include unpublished trials. Only 20% of reviews actually included data from unpublished trials. Further, only 23% of the reviews that did not include unpublished trials made any reference to the possibility of publication bias. These results suggest that a minority of Cochrane reviews include unpublished trials and that the possibility of publication bias could pose a real threat to their validity, yet this threat is rarely discussed within reviews. We suggest that even where resources do not permit collecting data from unpublished trials, every review should attempt to identify such trials in order to set their review in context. Where no effort is made to identify unpublished trials, this should be stated explicitly in the review and the possibility of publication bias discussed. The issue of potential publication bias needs to be debated within the collaboration and individuals and groups share and develop their strategies for seeking and obtaining information from unpublished trials.
Methods: A sample was obtained by selecting all completed reviews listed as new items in the Cochrane Library 2000, Issue 1. Reviews were scored as to whether their search strategies specifically mentioned a. any search method more extensive than electronic bibliographic searches and searches of their CRG specialist register, b. communicating directly with trialists etc. c. seeking unpublished trials. The number of trials included in each review that were a. published as full papers, b. published as abstracts c. unpublished were counted. For each review a text search using the phrase "publication bias" was done and outcome summarised.
Results: Of 52 completed systematic reviews listed as new items, 71% reported search strategies more extensive than bibliographic searching, 75% reported communication with trialists etc. and 52% specifically mentioned seeking unpublished trials. Of the 51 systematic reviews that identified eligible trials, 59% included only published trials, 29% included trials reported as abstracts or letters and 20% included unpublished trials. Of the 10 reviews that included unpublished trials, half reported a search strategy seeking such trials while the other half made no mention of looking for unpublished trials in their search strategy. 12 reviews used the phrase publication bias of which 10 planned to use funnel plots to assess the possibility of publication bias. Owing to inadequate or inappropriate data, only 4 were able to do so.
Discussion: Only 52% of the sample of Cochrane systematic reviews reported actively searching for unpublished trials, although 75% reported asking a variety of people to help identify trials, which may implicitly include unpublished trials. Only 20% of reviews actually included data from unpublished trials. Further, only 23% of the reviews that did not include unpublished trials made any reference to the possibility of publication bias. These results suggest that a minority of Cochrane reviews include unpublished trials and that the possibility of publication bias could pose a real threat to their validity, yet this threat is rarely discussed within reviews. We suggest that even where resources do not permit collecting data from unpublished trials, every review should attempt to identify such trials in order to set their review in context. Where no effort is made to identify unpublished trials, this should be stated explicitly in the review and the possibility of publication bias discussed. The issue of potential publication bias needs to be debated within the collaboration and individuals and groups share and develop their strategies for seeking and obtaining information from unpublished trials.