The prevalence of outcome reporting bias in trials (the ORBITstudy)

Article type
Authors
Kirkham J, Altman D, Dodd S, Dwan K, Gamble C, Jacoby A, Taylor S, Williamson P
Abstract
Background: A previous small study of five Cochrane reviews demonstrated that the selective non-reporting of outcomes within a study can have a substantial effect on meta-analysis when the amount of missing data is large (Williamson and Gamble, 2005). The authors concluded that a larger study of an unselected cohort was warranted. The ORBIT study (Outcome Reporting Bias In Trials) is examining the prevalence and impact of outcome reporting bias within a large cohort of systematic reviews. Objectives: To estimate the prevalence of outcome reporting bias in an unselected cohort of 297 Cochrane reviews. Methods: For each review, we identify whether all eligible trials have been included in the metaanalysis of the primary outcome and produce a matrix showing the reporting of all outcomes in each trial, distinguishing full/partial/no information. The primary reviewer is contacted, given information about our study, an outcome matrix for their review, and asked to participate by providing content expertise. For each trial not reporting the outcome of interest, the reviewer and research assistant independently scrutinise all publications relating to that trial, and classify, according to an established protocol, how likely it was that the outcome had been measured but not reported. Secondary relevant findings relating to the general quality of the review process are also sought for and discussed with the reviewer. Examples include reasons for missing protocols and changes in primary outcomes. Results: The ORBIT study investigates all new reviews published between Issue 4, 2006 and Issue 2, 2007 of The Cochrane Library. A total of 175/297 (59%) reviews were eligible for the study because not all included trials fully reported on the review primary outcome of interest. Of the 81 eligible reviews investigated thus far, a total of 735 trial reports have been scrutinised, 197 of which exhibited some form of outcome reporting bias. In 123/197 (62%) of cases, the level of suspicion of outcome reporting bias was deemed to be high. Conclusions: This presentation will include the results from the completed study, highlighting issues with the general reporting of outcomes and raising the awareness of outcome reporting bias.