Understanding the impact of outcome reporting bias in trials (the ORBIT study)

Dwan K, Altman D, Dodd S, Gamble C, Jacoby A, Kirkham J, Taylor S, Williamson P

Background: A previous small study of five Cochrane reviews demonstrated that the selective non-reporting of outcomes within a study can have a substantial effect on meta-analysis when the amount of missing data is large (Williamson and Gamble, 2005). The authors concluded a larger study of an unselected cohort was warranted. Objectives: To estimate the impact of outcome reporting bias (ORB) on the meta-analysis of the primary outcome of 297 reviews. To review the guidelines issued by organisations and charities that fund clinical trials with respect to this issue. Methods: The bound for the maximum bias approach was used to assess the impact of ORB to enable results to be comparable across reviews. The number of meta-analyses of the primary outcome in each review was also assessed. National and international organisations and charities were contacted to establish whether they fund clinical trials and if they issued guidelines to researchers. These were assessed to see what guidance was given regarding publication. Issues relating to publication bias and ORB were reviewed, including trial registration and protocol adherence. Results: To date, 20% of reviews had one meta-analysis of the primary outcome, 27% involved no meta-analysis, and 53% included more than one meta-analysis of the primary outcome. Sensitivity analyses will assess the robustness of the conclusions of the original analysis and results will be presented along with examples. Relatively little guidance is given by funding bodies with regard to publication, stating that work should be published in peer reviewed journals. Statements found in the guidelines generally refer to publication bias rather than ORB. Conclusions: Reviewers should scrutinise trials with missing outcome data and ensure that an attempt to contact trialists is made if the study does not report results. The lack of reporting of specified outcome(s) should not be an automatic reason for exclusion of studies. There is a need to provide more detailed guidance for those conducting and reporting clinical trials to help prevent the selective reporting of outcomes. Current guidelines require updating.