Article type
Year
Abstract
Background:
The Grading of Recommendations, Assessment, Development and Evaluation (GRADE) tool is often used to assess the quality of a body of evidence. However, it is currently unknown whether and how the GRADE tool can be used when conducting overviews of reviews (overviews) - an emerging method of evidence synthesis that compiles information from multiple systematic reviews (SRs).
Objectives:
To identify methodological considerations involved when using information from existing SRs to grade the outcomes included in an overview.
Methods:
We chose eight clinical areas and searched for all Cochrane and non-Cochrane SRs published in each topic area. Two reviewers independently conducted GRADE assessments for a subset of outcomes in the SRs and resolved discrepancies through consensus. Challenges surrounding the use of the GRADE tool were noted and discussed.
Results:
To date, we have assessed 111 outcomes in four clinical areas (acute otitis media, croup, bronchiolitis, eczema). There was moderate agreement between reviewers for final GRADE assessments, though agreement was higher when grading outcomes in Cochrane (kappa = 0.63) compared with non-Cochrane (kappa = 0.49) SRs. For outcomes presented in non-Cochrane SRs, we were unable to assess at least one GRADE domain between 6% to 15% of the time, and we were unable to provide a final GRADE assessment for 6% of outcomes. We were able to assess all four GRADE domains almost all of the time for Cochrane SRs.
Conclusions:
It was easier to grade quality of evidence for outcomes presented in Cochrane, as opposed to non-Cochrane, SRs, because Cochrane Reviews had more detailed reporting. For example, we could not assess the 'study limitations' domain without an adequate description of risk of bias of included studies, and we could not assess the 'consistency' domain without forest plots or measures of heterogeneity. Grading the quality of evidence at the overview level is often possible, but may be challenging. We present detailed descriptions, and potential solutions, for problems reviewers may encounter when applying GRADE on SRs that they have not conducted themselves.
The Grading of Recommendations, Assessment, Development and Evaluation (GRADE) tool is often used to assess the quality of a body of evidence. However, it is currently unknown whether and how the GRADE tool can be used when conducting overviews of reviews (overviews) - an emerging method of evidence synthesis that compiles information from multiple systematic reviews (SRs).
Objectives:
To identify methodological considerations involved when using information from existing SRs to grade the outcomes included in an overview.
Methods:
We chose eight clinical areas and searched for all Cochrane and non-Cochrane SRs published in each topic area. Two reviewers independently conducted GRADE assessments for a subset of outcomes in the SRs and resolved discrepancies through consensus. Challenges surrounding the use of the GRADE tool were noted and discussed.
Results:
To date, we have assessed 111 outcomes in four clinical areas (acute otitis media, croup, bronchiolitis, eczema). There was moderate agreement between reviewers for final GRADE assessments, though agreement was higher when grading outcomes in Cochrane (kappa = 0.63) compared with non-Cochrane (kappa = 0.49) SRs. For outcomes presented in non-Cochrane SRs, we were unable to assess at least one GRADE domain between 6% to 15% of the time, and we were unable to provide a final GRADE assessment for 6% of outcomes. We were able to assess all four GRADE domains almost all of the time for Cochrane SRs.
Conclusions:
It was easier to grade quality of evidence for outcomes presented in Cochrane, as opposed to non-Cochrane, SRs, because Cochrane Reviews had more detailed reporting. For example, we could not assess the 'study limitations' domain without an adequate description of risk of bias of included studies, and we could not assess the 'consistency' domain without forest plots or measures of heterogeneity. Grading the quality of evidence at the overview level is often possible, but may be challenging. We present detailed descriptions, and potential solutions, for problems reviewers may encounter when applying GRADE on SRs that they have not conducted themselves.