Are we evaluating methodological quality in the same way? Comparison of tools used in general health research and physical therapy with the 'Risk of bias' tool

Armijo-Olivo S1, Ospina M2, Rowe B3, Hartling L1
1Alberta Research Center for Health Evidence, Department of Pediatrics, University of Alberta, Canada, 2School of Public Health, University of Alberta, Canada, 3Faculty of Medicine and Dentistry, University of Alberta, Canada

Background: Assessing the methodological quality of randomized controlled trials (RCTs) is crucial to understand how bias may affect study results and to accurately identify treatment effects. A wide variety of tools are available to evaluate RCT quality; however, it is unknown how these tools compare in terms of the items included.

Objectives: To describe the frequency of individual items across quality assessment tools and to compare tools used in physical therapy (PT) to those used in general health research.

Methods: We identified tools used to evaluate the methodological quality of RCTs in general health research and PT following a comprehensive search and using a systematic approach. We extracted the items included in each tool. We calculated the frequency of each item used across tools and compared these among tools used in general health research and PT. Comparisons were also made with the Cochrane Risk of Bias (RofB) tool.

Results: Twenty-one tools were identified: 15 used in general health research and seven in PT. Ninety-five different items were used across tools. The most frequent items appearing in 15 of the general tools (67%) were: description of randomization, withdraws and dropouts, inclusion and exclusion criteria, and appropriate statistical analysis. In contrast, the most frequent items appearing in six out of seven tools used for PT (85.7%) were: baseline comparability, blinding of investigator/assessor, and use of intention-to-treat analysis. Prominent features of the RoB tool such as sequence generation and allocation concealment were included in four (57%) and five (71%) of the PT tools, respectively. Ten (63%) and six (40%) of the general health research tools included randomization and allocation concealment, respectively.

Conclusions: There is extensive variation in the items included across quality assessment tools. It is critical that methodological features associated with bias are captured in tools used for PT trials.