Article type
Year
Abstract
Background: We describe our experience of using a modified version of the risk of bias (RoB) tool for non-randomised comparative studies.
Objectives: - To assess inter-rater agreement for RoB assessment; - To assess time to complete RoB assessment; - To explore the association between RoB and treatment effect size.
Methods: Our sample comprised full text primary reports included in a systematic review comparing laparoscopic versus robotic prostatectomy for localised prostate cancer (n = 49). We excluded non-English language publications and conference abstracts. Three teams of two reviewers assessed individual categories as high, low or unclear RoB. Joint agreement was then reached for overall RoB. We explored differences in treatment effect size according to RoB for the outcome positive surgical margin (n = 9). Inter-rater agreement was assessed using the Kappa statistic. We weighted the analysis to reflect higher disagreement between high and low risk compared to high/low and unclear judgements. We timed individual RoB assessment and time to reach joint agreement. Results: Twenty five studies were judged as having high overall RoB, 13 were judged as low RoB and 11 were judged unclear. Standard and weighted Kappa values for inter-rater agreement were 0.34 and 0.35 respectively. The median (range) time for individual assessment was 30minutes (10 to 49minutes). The median time for reaching agreement between reviewers was 10 minutes (2 to 38minutes). The effect estimate for all studies was 0.61 (95% CI 0.46 to 0.83). The effect estimate for low risk studies was 0.73 (95% CI 0.29 to 1.75).
Conclusions: We achieved fair agreement between reviewers for RoB assessment of non-randomised studies. Although the process was time consuming, using a modified version of the Cochrane (RoB) tool proved useful for demonstrating conservative effect estimates in our systematic review. We suggest including more RoB levels and further validation could improve inter-rater agreement.
Objectives: - To assess inter-rater agreement for RoB assessment; - To assess time to complete RoB assessment; - To explore the association between RoB and treatment effect size.
Methods: Our sample comprised full text primary reports included in a systematic review comparing laparoscopic versus robotic prostatectomy for localised prostate cancer (n = 49). We excluded non-English language publications and conference abstracts. Three teams of two reviewers assessed individual categories as high, low or unclear RoB. Joint agreement was then reached for overall RoB. We explored differences in treatment effect size according to RoB for the outcome positive surgical margin (n = 9). Inter-rater agreement was assessed using the Kappa statistic. We weighted the analysis to reflect higher disagreement between high and low risk compared to high/low and unclear judgements. We timed individual RoB assessment and time to reach joint agreement. Results: Twenty five studies were judged as having high overall RoB, 13 were judged as low RoB and 11 were judged unclear. Standard and weighted Kappa values for inter-rater agreement were 0.34 and 0.35 respectively. The median (range) time for individual assessment was 30minutes (10 to 49minutes). The median time for reaching agreement between reviewers was 10 minutes (2 to 38minutes). The effect estimate for all studies was 0.61 (95% CI 0.46 to 0.83). The effect estimate for low risk studies was 0.73 (95% CI 0.29 to 1.75).
Conclusions: We achieved fair agreement between reviewers for RoB assessment of non-randomised studies. Although the process was time consuming, using a modified version of the Cochrane (RoB) tool proved useful for demonstrating conservative effect estimates in our systematic review. We suggest including more RoB levels and further validation could improve inter-rater agreement.