Risk of bias tool evaluation: process and results

Oral
2010 Keystone
Savovic J1, Altman D2, Higgins J3, Moher D4, Sterene J1, Turner L5, Weeks L4
1School of Social & Community Medicine, University of Bristol, Bristol, UK
2Centre for Statistics in Medicine, University of Oxford, Oxford, UK
3MRC Biostatistics Unit, University of Cambridge, Cambridge, UK
4Clinical Epidemiology, Ottawa Hospital Research Institute, Ottawa, Ontario, Canada
5Cochrane Bias Methods Group, OHRI, Ottawa, Ontario, Canada
Background: Version 5 of the Cochrane Handbook introduced the Risk of Bias (ROB) assessment tool, which represents a more comprehensive undertaking than previous quality assessments. There is a need to evaluate its implementation, examine barriers and facilitators to its use, and consider availability of training. Objectives: To obtain feedback from a range of stakeholders within the Cochrane Collaboration regarding their experiences with and perceptions of the RoB tool and associated guidance materials. Methods: We used qualitative and quantitative methods to evaluate the ROB tool. Four focus groups were held with international participants (n = 25). These followed a semi-structured format with a list of pre-specified topics, and were transcribed. Their results informed development of two questionnaires used in online surveys (distributed through established Cochrane mailing lists) of (a) review authors and (b) managing editors and other review group staff. We enquired about experience and perceptions of the ROB tool, incorporation of ROB assessments in meta-analyses, and training requirements. Results: We received 190 responses from authors who had used the RoB tool, 132 from authors who had not, and 58 from review group staff. RoB assessments take, on average, 10–60 minutes per study to complete: 83% of respondents deemed this acceptable. Most respondents thought that RoB assessments were better than past approaches to trial quality assessment. Most authors liked the standardized approach (81%) and the ability to provide quotes to support judgments (74%). About a third of participants did not like the increased workload, and found the wording describing judgments of ROB to be unclear. Most authors (75%) thought availability of training materials was sufficient, but many expressed an interest in online training. Conclusions: Overall, respondents identified positive experiences and perceptions of the RoB tool. Revisions of the tool and associated guidance, and improved provision of training, may improve implementation.