Article type
Year
Abstract
Background:
AMSTAR (A MeaSurement Tool to Assess systematic Reviews) is an 11-item tool that has been used frequently to check the quality of a systematic review and determine whether the most important elements are reported. The recently developed ROBIS (Risk of Bias in Systematic Reviews) tool was designed to evaluate the level of bias present within a systematic review.
Objectives:
The aim of this work was to compare AMSTART and ROBIS as part of an overview of systematic reviews of various complementary and alternative medicine (CAM) for fibromyalgia with regard to content, inter-rater reliability and usability.
Methods:
We searched five databases - MEDLINE, Embase, AMED (via OVID), Web of Science and CENTRAL - from their inception to December 2015. Two reviewers independently assessed each review using both tools. The inter-rater reliability of all sub-sections and the overall ratings using each instrument (AMSTAR and ROBIS) were calculated using the unweighted kappa statistic and percentage agreement.
Results:
We included 15 systematic reviews in this overview. In general, there was a high level of consistency between the overall AMSTAR scale and the ROBIS risk of bias rating. There were discrepancies with three reviews. Overall, five reviews scored 6 or above on the AMSTAR scale. The inter-rater agreement was good (Ƙ = 0.70, 95% confidence interval (CI) 0.60 to 0.85), with 83.6% agreement between two raters. Using the ROBIS tool, seven reviews achieved a low risk of bias rating, six a high rating and two were rated as having an unclear risk of bias. The inter-rater agreement was fair (Ƙ = 0.32, 95% CI 0.13 to 0.48), with 60.0% agreement between the two raters.
Conclusions:
ROBIS is an effective tool for assessing risk of bias in systematic reviews, but the agreement was less good and it is more difficult to use compared to AMSTAR. Reviews that included a meta-analysis were easier to rate; however, further developmental work could improve its use in reviews without a formal synthesis.
Patient or healthcare consumer involvement:
No patients or healthcare consumers were involved in this piece of methodological work.
AMSTAR (A MeaSurement Tool to Assess systematic Reviews) is an 11-item tool that has been used frequently to check the quality of a systematic review and determine whether the most important elements are reported. The recently developed ROBIS (Risk of Bias in Systematic Reviews) tool was designed to evaluate the level of bias present within a systematic review.
Objectives:
The aim of this work was to compare AMSTART and ROBIS as part of an overview of systematic reviews of various complementary and alternative medicine (CAM) for fibromyalgia with regard to content, inter-rater reliability and usability.
Methods:
We searched five databases - MEDLINE, Embase, AMED (via OVID), Web of Science and CENTRAL - from their inception to December 2015. Two reviewers independently assessed each review using both tools. The inter-rater reliability of all sub-sections and the overall ratings using each instrument (AMSTAR and ROBIS) were calculated using the unweighted kappa statistic and percentage agreement.
Results:
We included 15 systematic reviews in this overview. In general, there was a high level of consistency between the overall AMSTAR scale and the ROBIS risk of bias rating. There were discrepancies with three reviews. Overall, five reviews scored 6 or above on the AMSTAR scale. The inter-rater agreement was good (Ƙ = 0.70, 95% confidence interval (CI) 0.60 to 0.85), with 83.6% agreement between two raters. Using the ROBIS tool, seven reviews achieved a low risk of bias rating, six a high rating and two were rated as having an unclear risk of bias. The inter-rater agreement was fair (Ƙ = 0.32, 95% CI 0.13 to 0.48), with 60.0% agreement between the two raters.
Conclusions:
ROBIS is an effective tool for assessing risk of bias in systematic reviews, but the agreement was less good and it is more difficult to use compared to AMSTAR. Reviews that included a meta-analysis were easier to rate; however, further developmental work could improve its use in reviews without a formal synthesis.
Patient or healthcare consumer involvement:
No patients or healthcare consumers were involved in this piece of methodological work.