Article type
Year
Abstract
Background: Selective reporting is included as a core domain of Cochrane’s tool for assessing risk of bias in randomised trials. There has been no evaluation of review authors’ use of this domain.
Objective: We aimed to evaluate assessments of selective reporting in a cross-section of Cochrane Reviews, and to outline areas for improvement.
Methods: We obtained data on selective reporting judgements for 8434 studies included in 586 Cochrane Reviews published in the Cochrane Database of Systematic Reviews from Issue 1-8, 2015. One author classified reasons for judgements of high risk of selective reporting bias. We randomly selected 100 reviews with at least one trial rated at high risk of outcome non-reporting bias (non-/partial reporting of an outcome on the basis of its results). One author recorded whether authors of these reviews incorporated the selective reporting assessments when interpreting results.
Results: We rated 1055 (13%) of the 8434 studies as being at high risk of selective reporting bias. The most common reason for a high risk judgement was concern about outcome non-reporting bias. Few studies were rated at high risk because of concerns about bias in selection of the reported result (e.g. reporting of only a subset of measurements, analysis methods or subsets of the data that were prespecified). Review authors did not always specify in the 'Risk of bias' tables the study outcomes that were not reported (84% of studies) or partially reported (61% of studies). At least one study was rated at high risk of outcome non-reporting bias in 31% of reviews. However, only 30% incorporated this information when interpreting results, by acknowledging that the synthesis of an outcome was missing data that were not/partially reported.
Conclusion: Our audit of user practice suggests that the assessment of selective reporting in the current risk of bias tool does not work well. It is not always clear which outcomes were selectively reported, or what the corresponding risk of bias is in the synthesis with missing outcome data. New tools that will make it easier for reviewers to convey this information are being developed.
Objective: We aimed to evaluate assessments of selective reporting in a cross-section of Cochrane Reviews, and to outline areas for improvement.
Methods: We obtained data on selective reporting judgements for 8434 studies included in 586 Cochrane Reviews published in the Cochrane Database of Systematic Reviews from Issue 1-8, 2015. One author classified reasons for judgements of high risk of selective reporting bias. We randomly selected 100 reviews with at least one trial rated at high risk of outcome non-reporting bias (non-/partial reporting of an outcome on the basis of its results). One author recorded whether authors of these reviews incorporated the selective reporting assessments when interpreting results.
Results: We rated 1055 (13%) of the 8434 studies as being at high risk of selective reporting bias. The most common reason for a high risk judgement was concern about outcome non-reporting bias. Few studies were rated at high risk because of concerns about bias in selection of the reported result (e.g. reporting of only a subset of measurements, analysis methods or subsets of the data that were prespecified). Review authors did not always specify in the 'Risk of bias' tables the study outcomes that were not reported (84% of studies) or partially reported (61% of studies). At least one study was rated at high risk of outcome non-reporting bias in 31% of reviews. However, only 30% incorporated this information when interpreting results, by acknowledging that the synthesis of an outcome was missing data that were not/partially reported.
Conclusion: Our audit of user practice suggests that the assessment of selective reporting in the current risk of bias tool does not work well. It is not always clear which outcomes were selectively reported, or what the corresponding risk of bias is in the synthesis with missing outcome data. New tools that will make it easier for reviewers to convey this information are being developed.