Article type
Year
Abstract
Background: The Risk of Bias (RoB) tool is an emerging method for determining the risk of bias of health trials. Recent research have recommended further testing of the psychometric properties of the RoB domains, and validating the tool in a wider range of research fields. Inter-rater reliability of the RoB tool has been evaluated for child trials, general medical and nursing trials. The inter-rater reliability of the RoB has not been evaluated for physical therapy (PT) trials.
Objectives: To test the inter-rater reliability of the RoB tool applied to PT trials by comparing ratings from Cochrane Review authors and blinded external reviewers.
Methods: Randomized controlled trials (RCTs) in PT were identified by searching the Cochrane Database of Systematic Reviews for meta-analysis of PT interventions. RoB assessments of PT trials included in the meta-analyses were conducted independently by two reviewers blinded to the RoB ratings reported in the Cochrane Reviews. Consensus ratings between the two reviewers were compared with the Cochrane RoB ratings. Agreement between Cochrane and blinded external reviewers for individual domains and the final rating of the RoB tool was assessed using weighted kappa (K) for categorical data (K = 0.0–0.40 poor; K = 0.41–0.60 moderate; K = 0.61–0.80 substantial).
Results: In total, 109 trials included in 17 Cochrane Reviews were assessed. Interrater agreement on the overall RoB rating was poor (K = 0.07). Agreement on individual domains of the RoB tool was poor (median K = 0.20) ranging from K = 0.01 (‘Other bias’) to K = 0.63 (‘Sequence generation’).
Conclusions: Risk of bias assessments are not consistent across different research groups. Results have implications for decision making since different recommendations can be reached depending on the group analyzing the evidence. Improved guidelines to apply the RoB tool and revisions to the tool for different health areas are needed.
Objectives: To test the inter-rater reliability of the RoB tool applied to PT trials by comparing ratings from Cochrane Review authors and blinded external reviewers.
Methods: Randomized controlled trials (RCTs) in PT were identified by searching the Cochrane Database of Systematic Reviews for meta-analysis of PT interventions. RoB assessments of PT trials included in the meta-analyses were conducted independently by two reviewers blinded to the RoB ratings reported in the Cochrane Reviews. Consensus ratings between the two reviewers were compared with the Cochrane RoB ratings. Agreement between Cochrane and blinded external reviewers for individual domains and the final rating of the RoB tool was assessed using weighted kappa (K) for categorical data (K = 0.0–0.40 poor; K = 0.41–0.60 moderate; K = 0.61–0.80 substantial).
Results: In total, 109 trials included in 17 Cochrane Reviews were assessed. Interrater agreement on the overall RoB rating was poor (K = 0.07). Agreement on individual domains of the RoB tool was poor (median K = 0.20) ranging from K = 0.01 (‘Other bias’) to K = 0.63 (‘Sequence generation’).
Conclusions: Risk of bias assessments are not consistent across different research groups. Results have implications for decision making since different recommendations can be reached depending on the group analyzing the evidence. Improved guidelines to apply the RoB tool and revisions to the tool for different health areas are needed.