Article type
Abstract
Background:
Existing guidelines on the conduct of overviews of reviews and umbrella reviews recommend an assessment of the certainty of evidence, but provide limited guidance on ‘how to’ apply Grading of Recommendations Assessment, Development and Evaluation (GRADE) to such a complex evidence synthesis. We present a transparent approach to applying GRADE to an overview of reviews focused on diagnostic test accuracy.
Methods:
The approach was developed in an iterative and exploratory fashion following discussion with 11 methodologists/health services researchers. Considerations at each iteration of the approach were documented. Key principles were distilled based on the five GRADE domains for downgrading (risk of bias, inconsistency, imprecision, indirectness and publication bias) and two domains for upgrading (test outcome relations and large effect estimates, and residual plausible bias or confounding) as identified in the relevant GRADE guidelines, particularly those on test accuracy.
Results:
A general principles approach of applying the five domains of GRADE to an overview of reviews and arriving at an overall summary judgement for each outcome was developed. These methods were applied to an overview of reviews on novel molecular imaging for the staging of prostate cancer. This took into account additional factors including: dealing with both the primary study risk of bias as assessed by the reviews and the risk of bias of systematic reviews themselves, and statistical heterogeneity observed in meta-analyses. Challenges encountered by the review team were also discussed, for example avoiding double penalising for factors already considered in the risk of bias assessment of the systematic reviews.
Conclusions:
While there are challenges to applying GRADE certainty of evidence to complex evidence syntheses such as overviews, the use of a consistent and clearly documented approach can support transparency and confidence in the conclusions that are drawn.
Statement on the relevance and importance:
This approach allowed us to apply GRADE to a complex body of evidence. The approach could be adapted by researchers or methodologists carrying out overviews of reviews or umbrella reviews on other areas of health services research.
Existing guidelines on the conduct of overviews of reviews and umbrella reviews recommend an assessment of the certainty of evidence, but provide limited guidance on ‘how to’ apply Grading of Recommendations Assessment, Development and Evaluation (GRADE) to such a complex evidence synthesis. We present a transparent approach to applying GRADE to an overview of reviews focused on diagnostic test accuracy.
Methods:
The approach was developed in an iterative and exploratory fashion following discussion with 11 methodologists/health services researchers. Considerations at each iteration of the approach were documented. Key principles were distilled based on the five GRADE domains for downgrading (risk of bias, inconsistency, imprecision, indirectness and publication bias) and two domains for upgrading (test outcome relations and large effect estimates, and residual plausible bias or confounding) as identified in the relevant GRADE guidelines, particularly those on test accuracy.
Results:
A general principles approach of applying the five domains of GRADE to an overview of reviews and arriving at an overall summary judgement for each outcome was developed. These methods were applied to an overview of reviews on novel molecular imaging for the staging of prostate cancer. This took into account additional factors including: dealing with both the primary study risk of bias as assessed by the reviews and the risk of bias of systematic reviews themselves, and statistical heterogeneity observed in meta-analyses. Challenges encountered by the review team were also discussed, for example avoiding double penalising for factors already considered in the risk of bias assessment of the systematic reviews.
Conclusions:
While there are challenges to applying GRADE certainty of evidence to complex evidence syntheses such as overviews, the use of a consistent and clearly documented approach can support transparency and confidence in the conclusions that are drawn.
Statement on the relevance and importance:
This approach allowed us to apply GRADE to a complex body of evidence. The approach could be adapted by researchers or methodologists carrying out overviews of reviews or umbrella reviews on other areas of health services research.