Article type
Abstract
"Background
The MeaSurement Tool to Assess Systematic Reviews (AMSTAR-2) is critical for assessing reviews including randomised or non-randomised studies of healthcare interventions, or both (1). With 16 items for evaluation (seven critical and nine noncritical), the discordances between reviewers are something to address with discussion and consensus, or a third reviewer, and demands time and effort. To identify the items with more disagreement can be useful to improve the tool.
Objectives
To evaluate the inter-rater reliability and the weighted kappa statistics of AMSTAR-2. To identify the items with less agreement.
Methods
We assessed the methodological quality with the AMSTAR-2 tool in the overview of systematic reviews about interventions to prevent adverse events in the intensive care unit. The study team was divided to evaluate 139 systematic reviews in pairs (2,3). We measured inter-rater and calculated Kappa weighted score for agreement between pairs and, by AMSTAR-2 items.
Results
In a preliminary analysis, agreement between reviewers was significantly high (94.1%) with a good strength of agreement (kw=.727, p<.001), been these results consistent with critical and noncritical items (93.2%, kw=.742, p<.001; and 89.9%, kw=.682, p<001, respectively). Critical items with the least agreement were those referring to the risk of bias and the assessment of heterogeneity in non-randomized studies (9.2 and 11.2), respectively. The non-critical items with the least agreement were the explanation of the justification for including the type of study design in systematic reviews and the detailed description of the studies (items 3 and 8).
Conclusions
Our results are in line with the AMSTAR-2 validation study (1).
The levels of agreement between the pairs of ratters varied across items, but they were moderate to substantial for most items. Differences between ratters reflect the demanding nature of some item-level judgments and should prompt group discussion of their causes and importance, and, if needed, consultation with experts in subject matter and methods. Prior training of the reviewers in the AMSTAR-2 instrument is necessary so that there is maximum consensus when applying it individually.
Statement on the relevance and importance to patients: This work will improve overview methods. Therefore, evidence production will be more robust.
"
The MeaSurement Tool to Assess Systematic Reviews (AMSTAR-2) is critical for assessing reviews including randomised or non-randomised studies of healthcare interventions, or both (1). With 16 items for evaluation (seven critical and nine noncritical), the discordances between reviewers are something to address with discussion and consensus, or a third reviewer, and demands time and effort. To identify the items with more disagreement can be useful to improve the tool.
Objectives
To evaluate the inter-rater reliability and the weighted kappa statistics of AMSTAR-2. To identify the items with less agreement.
Methods
We assessed the methodological quality with the AMSTAR-2 tool in the overview of systematic reviews about interventions to prevent adverse events in the intensive care unit. The study team was divided to evaluate 139 systematic reviews in pairs (2,3). We measured inter-rater and calculated Kappa weighted score for agreement between pairs and, by AMSTAR-2 items.
Results
In a preliminary analysis, agreement between reviewers was significantly high (94.1%) with a good strength of agreement (kw=.727, p<.001), been these results consistent with critical and noncritical items (93.2%, kw=.742, p<.001; and 89.9%, kw=.682, p<001, respectively). Critical items with the least agreement were those referring to the risk of bias and the assessment of heterogeneity in non-randomized studies (9.2 and 11.2), respectively. The non-critical items with the least agreement were the explanation of the justification for including the type of study design in systematic reviews and the detailed description of the studies (items 3 and 8).
Conclusions
Our results are in line with the AMSTAR-2 validation study (1).
The levels of agreement between the pairs of ratters varied across items, but they were moderate to substantial for most items. Differences between ratters reflect the demanding nature of some item-level judgments and should prompt group discussion of their causes and importance, and, if needed, consultation with experts in subject matter and methods. Prior training of the reviewers in the AMSTAR-2 instrument is necessary so that there is maximum consensus when applying it individually.
Statement on the relevance and importance to patients: This work will improve overview methods. Therefore, evidence production will be more robust.
"