Article type
Year
Abstract
Background: Evaluation of internal validity of randomized controlled trials (RCTs) included in Cochrane systematic reviews (CSR) implies critical assessment of the risk of bias (RoB) in the studies' results, and is regarded an essential component of a review process. The Cochrane RoB tool comprises judgments of the level of RoB as low, unclear or high for seven different domains, with accompanying explanations for each judgment. It is recommended that it be used by two authors independently, a content expert and a methodologist.
Objectives: The aim of this study was to investigate the inter-rater reliability in assessing RoB in RCTs included in CSRs among review authors with different methodological and clinical backgrounds.
Methods: Three CSRs on interdental cleaning, one of which is an ongoing review, have been conduced with 44 included RCTs. Three review authors enrolled in all three CSRs, two content experts (TMJ, TPP), and a methodologist (DS), performed independent RoB assessments using the Cochrane RoB Tool for 31 out of 44 RCTs. Level of agreement in RoB assessments between review authors was analysed using the Advanced Analytics LLC software for Excel Windows (AGREESTAT 2011, Advanced Analytics, Maryland, USA) for calculating the Cohen kappa statistics.
Results: Preliminary results showed moderate agreement between a methodologist and both content experts, with = 0.50329, standard error (SE) = 0.05224 (95% CI 0.4 to 0.606) and the percent of agreement of 0.71 (95% CI 0.651 to 0.775) between DS and TMJ, and = 0.51280, SE = 0.050568 (95% CI 0.413 to 0.612), the percent of agreement of 0.71770, SE = 0.03121 (95% CI 0.656 to 0.779) between DS and TPP. Substantial agreement was found between the two content experts, with = 0.78248, SE = 0.039729 (95% CI 0.704 to 0.861) and the percentage of agreement of 0.87560, SE = 0.022884 (95% CI 0.83 to 0.921).
Conclusions: Content experts have different opinions about RoB compared to a methodology expert. Different opinions are beneficial as they provide a different perspective on the potential source and impact of bias influencing the results of RCTs and subsequently the findings of a CSR.
Patient or healthcare consumer involvement: This research has been performed by Cochrane members. Dental, especially oral hygiene topics are specific because we can all relate to them as patients or consumers.
Objectives: The aim of this study was to investigate the inter-rater reliability in assessing RoB in RCTs included in CSRs among review authors with different methodological and clinical backgrounds.
Methods: Three CSRs on interdental cleaning, one of which is an ongoing review, have been conduced with 44 included RCTs. Three review authors enrolled in all three CSRs, two content experts (TMJ, TPP), and a methodologist (DS), performed independent RoB assessments using the Cochrane RoB Tool for 31 out of 44 RCTs. Level of agreement in RoB assessments between review authors was analysed using the Advanced Analytics LLC software for Excel Windows (AGREESTAT 2011, Advanced Analytics, Maryland, USA) for calculating the Cohen kappa statistics.
Results: Preliminary results showed moderate agreement between a methodologist and both content experts, with = 0.50329, standard error (SE) = 0.05224 (95% CI 0.4 to 0.606) and the percent of agreement of 0.71 (95% CI 0.651 to 0.775) between DS and TMJ, and = 0.51280, SE = 0.050568 (95% CI 0.413 to 0.612), the percent of agreement of 0.71770, SE = 0.03121 (95% CI 0.656 to 0.779) between DS and TPP. Substantial agreement was found between the two content experts, with = 0.78248, SE = 0.039729 (95% CI 0.704 to 0.861) and the percentage of agreement of 0.87560, SE = 0.022884 (95% CI 0.83 to 0.921).
Conclusions: Content experts have different opinions about RoB compared to a methodology expert. Different opinions are beneficial as they provide a different perspective on the potential source and impact of bias influencing the results of RCTs and subsequently the findings of a CSR.
Patient or healthcare consumer involvement: This research has been performed by Cochrane members. Dental, especially oral hygiene topics are specific because we can all relate to them as patients or consumers.