Article type
Year
Abstract
Background:
Cochrane Reviews often include two graphs to summarize overall risk of bias (RoB) of included studies: a horizontal bar chart for RoB of all included studies per domain assessed (RoB graph) and a figure with review authors’ judgement for each domain and study (RoB summary). Both use the tricolour code for low, unclear and high RoB in a body of evidence but do not account for different information content of included studies (e.g. study size, number of events or variance) or consider which study sub-sets are used in meta-analyses (MA) for each outcome.
Objectives:
We aimed to explore how weighting (e.g. for study size or inverse variance from MA) change RoB graphs.
Methods: We created a MS Excel tool allowing weighting when generating RoB graphs. We used published Cochrane Reviews to illustrate how two alternative approaches change RoB graphs: 1) creating a single RoB graph with length of the tricolour bars weighted for relative study size (% of total no of participants) and 2) creating different RoB graphs per outcome using the inverse-variance weights from MA.
Results:
For illustration, we used the 2011 four-included study Cochrane Review by New (CD004214) with 14, 182, 60, and 94 participants (4%, 52%, 17%, and 27% of all 350 participants). The published RoB graph is displayed in Figure 1A. Weighting for relative study size resulted in Figure 1B. Using inverse-variance weights (92.8% and 7.2%) from the MA for one primary outcome (daily weight gain) including two studies resulted in Figure 1C.
Conclusions:
For this body of evidence the modified graphs conveyed a different picture of RoB. In some instances standard RoB graphs provide an overly pessimistic picture because small studies with higher RoB have equal weight. Introducing a weighting factor such as study size may help counteract the overestimation of RoB in standard graphs. Drawing separate RoB graphs for review outcomes (e.g. those in 'Summary of findings' tables) allows inverse-variance weighting in RoB graphs consistent with forest plots and may improve their use e.g. in GRADE RoB assessments. We will discuss additional examples and dis/advantages of alternative approaches.
Cochrane Reviews often include two graphs to summarize overall risk of bias (RoB) of included studies: a horizontal bar chart for RoB of all included studies per domain assessed (RoB graph) and a figure with review authors’ judgement for each domain and study (RoB summary). Both use the tricolour code for low, unclear and high RoB in a body of evidence but do not account for different information content of included studies (e.g. study size, number of events or variance) or consider which study sub-sets are used in meta-analyses (MA) for each outcome.
Objectives:
We aimed to explore how weighting (e.g. for study size or inverse variance from MA) change RoB graphs.
Methods: We created a MS Excel tool allowing weighting when generating RoB graphs. We used published Cochrane Reviews to illustrate how two alternative approaches change RoB graphs: 1) creating a single RoB graph with length of the tricolour bars weighted for relative study size (% of total no of participants) and 2) creating different RoB graphs per outcome using the inverse-variance weights from MA.
Results:
For illustration, we used the 2011 four-included study Cochrane Review by New (CD004214) with 14, 182, 60, and 94 participants (4%, 52%, 17%, and 27% of all 350 participants). The published RoB graph is displayed in Figure 1A. Weighting for relative study size resulted in Figure 1B. Using inverse-variance weights (92.8% and 7.2%) from the MA for one primary outcome (daily weight gain) including two studies resulted in Figure 1C.
Conclusions:
For this body of evidence the modified graphs conveyed a different picture of RoB. In some instances standard RoB graphs provide an overly pessimistic picture because small studies with higher RoB have equal weight. Introducing a weighting factor such as study size may help counteract the overestimation of RoB in standard graphs. Drawing separate RoB graphs for review outcomes (e.g. those in 'Summary of findings' tables) allows inverse-variance weighting in RoB graphs consistent with forest plots and may improve their use e.g. in GRADE RoB assessments. We will discuss additional examples and dis/advantages of alternative approaches.