Assessing the quality of a body of evidence: experiences with the GRADE approach

Tags: Oral
Langendam M, Kramer S, Elbers R, Hooft L, Scholten R

Background: The Cochrane Collaboration has adopted the Grades of Recommendation, Assessment, Development and Evaluation (GRADE) system for evaluating the quality of evidence for the main outcomes reported in a systematic review. GRADE specifies four quality levels (high, moderate, low and very low). The starting point is a high quality rating for a body of evidence based on randomized controlled trials (RCTs). This rating can be downgraded, depending on the presence of five factors: within-study risk of bias, directness of evidence, heterogeneity, imprecision of the effect estimates and risk of publication bias. Although over 20 organizations have adopted GRADE, the use of GRADE is relatively new and concrete examples of applying GRADE are not very common. Objectives: To describe the challenges that we encountered while applying GRADE in various projects and how we overcame these challenges. Methods: We used GRADE to assess the quality of evidence in an Overview of reviews on the effectiveness of physiotherapy for fibromyalgia and in an evidence-based guideline on exhaled nitric oxide monitoring in children with asthma. Other projects are planned. We kept a log of the problems we encountered and searched the literature to find solutions. Results: The problems we encountered were poor reporting of study quality (difficult to assess risk of bias) and results, absence of a meta-analysis and inclusion of only one RCT (difficult to assess inconsistency and imprecision). In the case of one RCT we decided to rate the quality of the evidence ‘low’ if there were no limitations in study design and ‘very low’ otherwise. For the other situations we decided not to use GRADE. We found different examples of how other authors addressed these difficulties. Conclusions: GRADE is a methodologically rigorous and user friendly system, but there are still some important questions to be answered to apply GRADE consistently and objectively.