Recommendations for updates to the Cochrane Collaboration’s Risk of Bias assessment tool, based on an evaluation of its initial implementation

Oral
2010 Keystone
Sterne J1, Altman D2, Higgins J3, Moher D4, Turner L5, Weeks L5
1School of Social & Community Medicine, University of Bristol, Bristol, UK
2Director of the Centre for Statistics in Medicine and Cancer Research UK Medical Statistics Group, University of Oxford, Oxford, UK
3MRC Biostatistics Unit, University of Cambridge, Cambridge, UK
4Epidemiology and Community Medicine, University of Ottawa, Ottawa, Ontario, Canada
5Cochrane Bias Methods Group, Ottawa Hospital Research Institute, Ottawa, Ontario, Canada
Background: Version 5 of the Cochrane Handbook introduced the Risk of Bias (ROB) assessment tool, which represents a more comprehensive undertaking than previous quality assessments. Objectives: To develop recommendations for changes to the ROB tool, based on an evaluation of its initial implementation. Methods: The Bias Methods Group, supported by the Cochrane Library Editorial Unit and funded by the Cochrane Opportunities Fund conducted an evaluation of the ROB tool. We used findings from focus groups to develop questionnaires that were used in online surveys of Cochrane stakeholders. A meeting of Cochrane methodologists, review authors, managing and coordinating editors, and Editorial Unit staff developed draft recommendations. Results: Overall, there was a positive response to the introduction of ROB assessments. Several areas for potential improvements were identified. The main recommendations arising from the evaluation include: • Immediate/short term: – Change wording of bias judgements from ‘‘yes/no’’ to ‘‘low/high risk of bias’’ – Introduce category headings for selection, performance and detection, attrition, reporting, and other bias – Manually split assessment of blinding into participants/personnel and outcome assessment – Clarify guidance, particularly for incomplete outcomes and selective outcome reporting, and assessment of ‘‘other sources of bias’’ – Produce clearer and more explicit guidance on decision making for incorporation of ROB assessments into meta-analyses. • Medium term (implementation with RevMan6 or later): – Structurally split assessment blinding into participants/personnel (performance bias heading) and blinding of outcome assessment (detection bias heading) –Weight ROB graphs by study size – Provide an algorithm for reaching a summary assessment of ROB per study/outcome – Develop online guidance and training materials including an online FAQ bank and examples of assessments. Conclusions: We hope that these changes will make it easier for authors to use the ROB tool, improve the reliability of assessments and improve the quality of Cochrane reviews.