Article type
Year
Abstract
Background: Cochrane systematic reviews (SR) must include an assessment of the risk of bias (RoB) of included studies. Although there are numerous tools to assess the RoB of any given study, the Cochrane Collaboration endorses the RoB tool since 2008. Despite this standardization effort, the tool is based on subjective judgments and requires access to all relevant methodological information of each included study.
Objectives: To assess whether different Cochrane Reviews assessing the same included studies agree in their use of RoB criteria.
Methods: We entered all Cochrane Reviews and their corresponding included studies into the Epistemonikos database (www.epistemonikos.org). Epistemonikos links SRs to all their included studies. We selected a random sample of studies included in two or more Cochrane SR published since 2008. For each included study we extracted the RoB assessment provided in the corresponding section of the Sr. This yielded at least two RoB assessments per study. Finally, we calculated agreements between the assessments made by different reviews for the same RoB item.
Results: Our search was conducted in January 2013. We identified 5765 studies included in 2 or more Cochrane SRs from which we selected a random sample of 40 records. The RoB item most consistently assessed was ‘randomization’, assessed in all studies. For 33 studies (82.5%) its assessment was concordant. Concordance was observed in 72.5% of studies for ‘allocation concealment’, and 67.5% for ‘attrition bias’. On the other side of the spectrum, concordance was only 20% for performance bias and detection bias. Complete results, including a larger sample will be presented at the Colloquium.
Conclusions: The Cochrane Collaboration has made efforts to standardize risk of bias assessment of included studies across SRs. However, our analysis shows that RoB assessment is still inconsistent and might lead to equivocal appraisal of the available evidence.
Objectives: To assess whether different Cochrane Reviews assessing the same included studies agree in their use of RoB criteria.
Methods: We entered all Cochrane Reviews and their corresponding included studies into the Epistemonikos database (www.epistemonikos.org). Epistemonikos links SRs to all their included studies. We selected a random sample of studies included in two or more Cochrane SR published since 2008. For each included study we extracted the RoB assessment provided in the corresponding section of the Sr. This yielded at least two RoB assessments per study. Finally, we calculated agreements between the assessments made by different reviews for the same RoB item.
Results: Our search was conducted in January 2013. We identified 5765 studies included in 2 or more Cochrane SRs from which we selected a random sample of 40 records. The RoB item most consistently assessed was ‘randomization’, assessed in all studies. For 33 studies (82.5%) its assessment was concordant. Concordance was observed in 72.5% of studies for ‘allocation concealment’, and 67.5% for ‘attrition bias’. On the other side of the spectrum, concordance was only 20% for performance bias and detection bias. Complete results, including a larger sample will be presented at the Colloquium.
Conclusions: The Cochrane Collaboration has made efforts to standardize risk of bias assessment of included studies across SRs. However, our analysis shows that RoB assessment is still inconsistent and might lead to equivocal appraisal of the available evidence.