Article type
Year
Abstract
Background: Overviews of reviews (overviews) compile information from multiple systematic reviews (SRs) to provide a single synthesis of relevant evidence for decision-making. It is recommended that authors assess and report the methodological quality of SRs in overviews--for example, using 'A MeaSurement Tool to Assess systematic Reviews' (AMSTAR). Currently, there is variation in whether and how overview authors assess and report SR quality, and limited guidance is available.
Objectives: To examine methodological considerations involved in using AMSTAR to assess quality of SRs in overviews, and to examine the impact of using an AMSTAR threshold (quality 'cutoff') as an inclusion criterion.
Methods: We selected a sample of seven overviews and searched for all SRs meeting each overview's inclusion criteria. Ninety-six SRs were included (30 Cochrane, 66 non-Cochrane). For each SR, two reviewers independently conducted AMSTAR assessments with consensus and discussed challenges encountered. We also extracted the main result and conclusion from each SR.
Results: Mean AMSTAR scores (/11) were significantly higher for Cochrane compared to non-Cochrane SRs (9.6 vs 5.5; P < 0.001). Mean inter-rater reliability was high overall, but was significantly higher for Cochrane SRs compared to non-Cochrane SRs (AC1 statistic: 0.84 vs 0.69; P = 0.002). Four challenges (and solutions) were identified when assessing AMSTAR in the context of overviews. We found no evidence that AMSTAR scores were correlated with the results or conclusions of Cochrane or non-Cochrane SRs.
Conclusions: High inter-rater reliability suggests that AMSTAR can be used successfully in overviews that include both Cochrane and non-Cochrane SRs, though minor modifications may be helpful. Cochrane SRs are often high quality and should be included in overviews, whereas non-Cochrane SRs with low AMSTAR scores may be challenging to use in overviews. A minimum AMSTAR score may be a useful inclusion criterion for overviews and should not introduce bias into the overview process since AMSTAR scores are not correlated with results or conclusions of SRs.
Objectives: To examine methodological considerations involved in using AMSTAR to assess quality of SRs in overviews, and to examine the impact of using an AMSTAR threshold (quality 'cutoff') as an inclusion criterion.
Methods: We selected a sample of seven overviews and searched for all SRs meeting each overview's inclusion criteria. Ninety-six SRs were included (30 Cochrane, 66 non-Cochrane). For each SR, two reviewers independently conducted AMSTAR assessments with consensus and discussed challenges encountered. We also extracted the main result and conclusion from each SR.
Results: Mean AMSTAR scores (/11) were significantly higher for Cochrane compared to non-Cochrane SRs (9.6 vs 5.5; P < 0.001). Mean inter-rater reliability was high overall, but was significantly higher for Cochrane SRs compared to non-Cochrane SRs (AC1 statistic: 0.84 vs 0.69; P = 0.002). Four challenges (and solutions) were identified when assessing AMSTAR in the context of overviews. We found no evidence that AMSTAR scores were correlated with the results or conclusions of Cochrane or non-Cochrane SRs.
Conclusions: High inter-rater reliability suggests that AMSTAR can be used successfully in overviews that include both Cochrane and non-Cochrane SRs, though minor modifications may be helpful. Cochrane SRs are often high quality and should be included in overviews, whereas non-Cochrane SRs with low AMSTAR scores may be challenging to use in overviews. A minimum AMSTAR score may be a useful inclusion criterion for overviews and should not introduce bias into the overview process since AMSTAR scores are not correlated with results or conclusions of SRs.