Article type
Year
Abstract
Background: assessing the risk of bias (RoB) in included studies is one of the key methodological aspects of systematic reviews. Cochrane Systematic Reviews appraise RoB of randomised controlled trials (RCTs) with the Cochrane RoB tool. Detailed instructions for using the Cochrane RoB tool are provided in the Cochrane Handbook for Systematic Reviews of Interventions (the Cochrane Handbook).
Objectives: to analyze adequacy of judgments about RoB for random sequence generation, and blinding of participants and personnel (performance bias) in Cochrane Systematic Reviews of randomized controlled trials (RCTs).
Methods: we extracted judgments and supporting comments for random sequence generation and performance bias from Cochrane Reviews’ RoB tables using automated data scraping. We parsed all intervention descriptions, judgments about risk of performance bias, and comments supporting judgments into simple categories. We assessed adequacy of RoB judgments against recommendations from the Cochrane Handbook.
Results: for analysis related to random sequence generation, we analysed 10,103 RCTs that were included in 704 Cochrane Reviews. For 5706 RCTs, randomisation was not described, but for the remaining RCTs, it was indicated that randomisation was performed using computer/software/internet (N = 2850), random number table (N = 883), mechanical method (N = 359) or it was incomplete/inappropriate (N = 305). Overall, 1220/10,103 trials (12%) did not have a RoB judgment in line with Cochrane Handbook guidance about randomisation. The highest proportion of misjudgements was found for trials with high RoB (28%), followed by those with low (20%) or unclear (3%) RoB. Therefore, one in eight judgments for the analysed domain in Cochrane Reviews was not in line with the Cochrane Handbook, and one in four of the judgments was 'high risk'.
For performance bias, we analyzed judgments of 10,429 RCTs included in 718 Cochrane Reviews. Overall, 1828 out of 6918 judgments (26%) for performance bias were not in line with the Cochrane Handbook and were therefore considered inadequate. In reviews where Cochrane authors have split the performance bias domain into two subdomains, based on blinded individuals, we found lower prevalence of inadequate risk of bias judgments, with 9% of judgements for blinding of participants, and 5.8% judgements for the blinding of personnel subdomain being judged inadequately.
Conclusion: in Cochrane Reviews, RoB assessments for random sequence generation, and blinding of participants and personnel were frequently not in line with Cochrane Handbook recommendations. Our results can help authors of both Cochrane and non-Cochrane reviews, which use the Cochrane RoB tool to avoid making common mistakes when assessing RoB in included trials. Interventions to improve these assessments should be taken in consideration.
Patient or healthcare consumer involvement: not applicable; this was a methodological study, and we did not include patients or healthcare consumers in the study.
Objectives: to analyze adequacy of judgments about RoB for random sequence generation, and blinding of participants and personnel (performance bias) in Cochrane Systematic Reviews of randomized controlled trials (RCTs).
Methods: we extracted judgments and supporting comments for random sequence generation and performance bias from Cochrane Reviews’ RoB tables using automated data scraping. We parsed all intervention descriptions, judgments about risk of performance bias, and comments supporting judgments into simple categories. We assessed adequacy of RoB judgments against recommendations from the Cochrane Handbook.
Results: for analysis related to random sequence generation, we analysed 10,103 RCTs that were included in 704 Cochrane Reviews. For 5706 RCTs, randomisation was not described, but for the remaining RCTs, it was indicated that randomisation was performed using computer/software/internet (N = 2850), random number table (N = 883), mechanical method (N = 359) or it was incomplete/inappropriate (N = 305). Overall, 1220/10,103 trials (12%) did not have a RoB judgment in line with Cochrane Handbook guidance about randomisation. The highest proportion of misjudgements was found for trials with high RoB (28%), followed by those with low (20%) or unclear (3%) RoB. Therefore, one in eight judgments for the analysed domain in Cochrane Reviews was not in line with the Cochrane Handbook, and one in four of the judgments was 'high risk'.
For performance bias, we analyzed judgments of 10,429 RCTs included in 718 Cochrane Reviews. Overall, 1828 out of 6918 judgments (26%) for performance bias were not in line with the Cochrane Handbook and were therefore considered inadequate. In reviews where Cochrane authors have split the performance bias domain into two subdomains, based on blinded individuals, we found lower prevalence of inadequate risk of bias judgments, with 9% of judgements for blinding of participants, and 5.8% judgements for the blinding of personnel subdomain being judged inadequately.
Conclusion: in Cochrane Reviews, RoB assessments for random sequence generation, and blinding of participants and personnel were frequently not in line with Cochrane Handbook recommendations. Our results can help authors of both Cochrane and non-Cochrane reviews, which use the Cochrane RoB tool to avoid making common mistakes when assessing RoB in included trials. Interventions to improve these assessments should be taken in consideration.
Patient or healthcare consumer involvement: not applicable; this was a methodological study, and we did not include patients or healthcare consumers in the study.