Article type
Year
Abstract
Background:
In Cochrane Reviews (CR) studies at a high or unclear risk of bias (RoB) are often down-graded or removed in sensitivity analyses, impacting on review conclusions. It is not clear whether review authors are consistent in their approach to these assessments.
Objectives:
To assess the consistency in RoB judgements across CR in the field of subfertility.
Methods:
84 reviews were identified which comprised 981 included studies. There were 46 studies that appeared in at least two reviews (duplicates). RoB judgements (low-risk/yes, unclear, high-risk/no) and supporting text were extracted where possible for each of the following domains: sequence generation, allocation concealment, blinding, incomplete outcome data and selective reporting.
Results:
Of the 46 duplicates, only 21 had judgements for all five bias domains, resulting in varying denominators for each domain (table 1). Overall, 46% of bias judgements differed between reviews. This inconsistency varied across domains; 29% of judgements on sequence generation differed compared to 65% of blinding assessments. Of interest is whether the inconsistencies change the judgement from low to unclear/high-risk (or vice-versa), and thus shift the study between subgroups in a sensitivity analyses. Overall, 35% were differences that would change the subgrouping (range 22% to 54% across domains). An explanation for differences may have been changing recommendations to authors over time. However, only 23% of differences were in reviews published using both Revman 4 and 5. Of all instances with different bias judgements, 27% supported their differing judgement with similar or identical reasoning; 73% of the time the support for the judgement was based on different information, potentially due to unmatched author correspondence, or a misunderstanding of either the bias domain or content from the papers.
Conclusions:
Here is a large amount of inconsistency in RoB judgements in subfertility reviews, the majority of which change the study from a low to unclear/high-risk of bias. Improved consistency across reviews could be achieved by updating and linking RoB judgements to studies within the CRS.
In Cochrane Reviews (CR) studies at a high or unclear risk of bias (RoB) are often down-graded or removed in sensitivity analyses, impacting on review conclusions. It is not clear whether review authors are consistent in their approach to these assessments.
Objectives:
To assess the consistency in RoB judgements across CR in the field of subfertility.
Methods:
84 reviews were identified which comprised 981 included studies. There were 46 studies that appeared in at least two reviews (duplicates). RoB judgements (low-risk/yes, unclear, high-risk/no) and supporting text were extracted where possible for each of the following domains: sequence generation, allocation concealment, blinding, incomplete outcome data and selective reporting.
Results:
Of the 46 duplicates, only 21 had judgements for all five bias domains, resulting in varying denominators for each domain (table 1). Overall, 46% of bias judgements differed between reviews. This inconsistency varied across domains; 29% of judgements on sequence generation differed compared to 65% of blinding assessments. Of interest is whether the inconsistencies change the judgement from low to unclear/high-risk (or vice-versa), and thus shift the study between subgroups in a sensitivity analyses. Overall, 35% were differences that would change the subgrouping (range 22% to 54% across domains). An explanation for differences may have been changing recommendations to authors over time. However, only 23% of differences were in reviews published using both Revman 4 and 5. Of all instances with different bias judgements, 27% supported their differing judgement with similar or identical reasoning; 73% of the time the support for the judgement was based on different information, potentially due to unmatched author correspondence, or a misunderstanding of either the bias domain or content from the papers.
Conclusions:
Here is a large amount of inconsistency in RoB judgements in subfertility reviews, the majority of which change the study from a low to unclear/high-risk of bias. Improved consistency across reviews could be achieved by updating and linking RoB judgements to studies within the CRS.