Background:

The Cochrane risk-of-bias tool for randomised trials is widely used. Evaluation of the tool in 2010 identified areas in which improvements could be made for ease of use. Some modifications were made in a minor revision in 2011, and the substantial suggestions have been addressed in a major update of the tool, known as RoB 2.0. Among the suggestions was a request that algorithms be developed to help review authors make judgments about risk of bias within each bias domain. Here we describe how we responded to this request and discuss some implications.

Objectives:

To develop algorithms to help users of the risk-of-bias tool for randomised trials reach risk of bias judgments for each domain.

Processes and outputs: In the development of RoB 2.0, working groups were formed for each domain of bias, and were first tasked with developing a series of signalling questions. These questions aim to elicit information about methods, observations and contexts likely to impact on risk of bias. The working groups were then asked to describe how answers to these questions would lead to a judgment about risk of bias. To facilitate implementation, we explored the use of formal-decision algorithms that directly map answers to suggested judgments. This proved challenging, but ultimately successful, and required several signalling questions to be revised so that they comprehensively covered the issues. All questions and algorithms were piloted and revised as appropriate.

Discussion and

conclusions:

We believe that automated algorithms reduce workload for systematic review authors, and should increase the consistency with which risk-of-bias judgments are made across reviews. We emphasise that the judgments proposed by the algorithms must be viewed only as suggested judgments, so they can be overridden by review authors. The algorithms map each combination of possible answers to signalling questions to a unique risk-of-bias judgment. In practice, it is not necessary to answer all the questions to uniquely determine a judgment. This raises the possibility that only a subset of the signalling questions need to be answered.