Article type
Year
Abstract
Background: Using GRADE (Grading of Recommendations Assessment, Development and Evaluation tool) to rate the quality of the body of evidence is complex and subjective, and how consistently reviewers apply the rating system is unclear.
Objectives: The aim of this study is to evaluate how changes in evidence contribute to changes in quality of evidence ratings by comparing Cochrane Reviews having quality of evidence tables with earlier and later versions.
Methods: We conducted a MEDLINE search of the Cochrane Library for systematic reviews with quality of evidence tables in sequential updates. For each outcome with a change in number of studies, sample size, or quality rating, we abstracted publication years, review authors, effect estimates, numbers of included studies, and sample sizes and conducted regression analyses to determine predictors of a change in quality rating.
Results: A total of 1814 systematic reviews were marked in the Cochrane Library as recently updated. Preliminary analysis of 350 reviews found nine that provided quality ratings tables for sequential updates, representing 36 different outcomes. Two factors associated with a change in quality rating in univariate analysis were: change in numbers of participants (P value 0.041) and change in magnitude of the effect estimates (P value 0.008). Changes in precision or statistical significance of effect estimates were not associated with a change in quality rating. Quality of the evidence was downgraded (n = 13) more often than upgraded (n = 4) and was sometimes downgraded with no change in the evidence.
Conclusions: Changes in the magnitude of effect estimates and numbers of participants were associated with changes in quality of the body of evidence ratings. Quality ratings were downgraded more often than upgraded which may indicate a reviewer 'experience' factor as the reasons for downgrading were not always obvious and it would be helpful if reviewers explained the rationale for these changes. Further analysis including the remainder of the reviews and the use of multivariable regression to explore relationships between variables is in progress.
Objectives: The aim of this study is to evaluate how changes in evidence contribute to changes in quality of evidence ratings by comparing Cochrane Reviews having quality of evidence tables with earlier and later versions.
Methods: We conducted a MEDLINE search of the Cochrane Library for systematic reviews with quality of evidence tables in sequential updates. For each outcome with a change in number of studies, sample size, or quality rating, we abstracted publication years, review authors, effect estimates, numbers of included studies, and sample sizes and conducted regression analyses to determine predictors of a change in quality rating.
Results: A total of 1814 systematic reviews were marked in the Cochrane Library as recently updated. Preliminary analysis of 350 reviews found nine that provided quality ratings tables for sequential updates, representing 36 different outcomes. Two factors associated with a change in quality rating in univariate analysis were: change in numbers of participants (P value 0.041) and change in magnitude of the effect estimates (P value 0.008). Changes in precision or statistical significance of effect estimates were not associated with a change in quality rating. Quality of the evidence was downgraded (n = 13) more often than upgraded (n = 4) and was sometimes downgraded with no change in the evidence.
Conclusions: Changes in the magnitude of effect estimates and numbers of participants were associated with changes in quality of the body of evidence ratings. Quality ratings were downgraded more often than upgraded which may indicate a reviewer 'experience' factor as the reasons for downgrading were not always obvious and it would be helpful if reviewers explained the rationale for these changes. Further analysis including the remainder of the reviews and the use of multivariable regression to explore relationships between variables is in progress.