Article type
Year
Abstract
Background: Well-designed randomized controlled trials (RCTs) are considered the 'gold standard' for evaluating an intervention's effectiveness. As the quality of them varies widely, a method is needed to separate higher from lower quality.
Objectives: To quantify the number of guidelines listed in the National Guideline Clearinghouse (NGC) that utilized a rating scheme to determine the quality of evidence and present a quantitative method to assess the quality of RCTs.
Methods: Data were abstracted from the NGC addressing systematic reviews. We reviewed the guideline matrix used by the NGC, which allowed for quantification of methods to assess the quality of the evidence.
Results: Of the systematic review-based guidelines (N = 2024) represented by the NGC, 25.3% (N = 513) do not utilize a weighting according to a rating scheme (scheme given) to assess the quality of the evidence; 2.8% (N = 56) use a rating scheme but do not provide further details (scheme not given), while 4.3% (N = 88) do not provide any methods regarding analysis of evidence. Further review found that many of those represented as having a rating scheme in the NGC largely use qualitative methods. A quantitative scoring method used by the American College of Occupational and Environmental Medicine (ACOEM) considers 11 criteria: randomization, concealed treatment allocation, baseline comparability, whether patient-, provider-, assessor blinded, controlled for co-interventions, compliance acceptable, dropout rate, timing of assessments equivalent, intention-to-treat analysis, and lack of bias. Each criterion is rated 0, 0.5, or 1.0. Study ratings range from 0-11. A study is considered to be low quality if the composite rating is 3.5 or less, moderate quality if rated 4-7.5, and high quality if rated 8-11. This system results in a testable article score and more reproducible guidelines methods.
Conclusion: Properly grading study quality and rating overall strength of evidence can produce improved levels of confidence about the scientific basis for guidelines.
Objectives: To quantify the number of guidelines listed in the National Guideline Clearinghouse (NGC) that utilized a rating scheme to determine the quality of evidence and present a quantitative method to assess the quality of RCTs.
Methods: Data were abstracted from the NGC addressing systematic reviews. We reviewed the guideline matrix used by the NGC, which allowed for quantification of methods to assess the quality of the evidence.
Results: Of the systematic review-based guidelines (N = 2024) represented by the NGC, 25.3% (N = 513) do not utilize a weighting according to a rating scheme (scheme given) to assess the quality of the evidence; 2.8% (N = 56) use a rating scheme but do not provide further details (scheme not given), while 4.3% (N = 88) do not provide any methods regarding analysis of evidence. Further review found that many of those represented as having a rating scheme in the NGC largely use qualitative methods. A quantitative scoring method used by the American College of Occupational and Environmental Medicine (ACOEM) considers 11 criteria: randomization, concealed treatment allocation, baseline comparability, whether patient-, provider-, assessor blinded, controlled for co-interventions, compliance acceptable, dropout rate, timing of assessments equivalent, intention-to-treat analysis, and lack of bias. Each criterion is rated 0, 0.5, or 1.0. Study ratings range from 0-11. A study is considered to be low quality if the composite rating is 3.5 or less, moderate quality if rated 4-7.5, and high quality if rated 8-11. This system results in a testable article score and more reproducible guidelines methods.
Conclusion: Properly grading study quality and rating overall strength of evidence can produce improved levels of confidence about the scientific basis for guidelines.