Article type
Year
Abstract
Background: GRADE (Grading of Recommendations, Assessment, Development and Evaluations) recently approved guidance for addressing the risk of bias associated with missing participant outcome data in meta-analyses. Thus far, however, application to examples has been limited.
Methods: We applied GRADE guidance to six systematic reviews published by our team and also re-assessed the risk of bias in six systematic reviews published by others; reviews included both dichotomous and continuous outcomes. The examples challenge the robustness of findings of statistically significant benefit; failure to establish benefit; statistically significant harm; and failure to establish statistically significant harm. We began with a primary meta-analysis using a complete case analysis, followed by sensitivity meta-analyses imputing, in each primary study, results for those with missing data. We then pooled across studies using the imputed data to determine the impact on the point of estimate and confidence interval. We applied progressively more stringent imputations.
Results: We found some examples robust to even the most stringent imputations (in which case we would not rate down for risk of bias); situations in which statistical significance was lost (if present), or observed (if absent), only for the most stringent assumptions (in which case one would rate down for risk of bias if one considered these stringent assumptions plausible); and situations in which statistical significance was lost if present, or observed if absent, in even less stringent imputations (in which case one would surely rate down for risk of bias). We observed instances in which application of our approach would lead to a decision to rate down for risk of bias when authors of the original systematic review concluded that missing data did not pose an important risk of bias problem.
Conclusion: This practical application of GRADE guidance documents the importance of the formal, structured evaluation of risk of bias due to missing data at the level of the meta-analysis.
Methods: We applied GRADE guidance to six systematic reviews published by our team and also re-assessed the risk of bias in six systematic reviews published by others; reviews included both dichotomous and continuous outcomes. The examples challenge the robustness of findings of statistically significant benefit; failure to establish benefit; statistically significant harm; and failure to establish statistically significant harm. We began with a primary meta-analysis using a complete case analysis, followed by sensitivity meta-analyses imputing, in each primary study, results for those with missing data. We then pooled across studies using the imputed data to determine the impact on the point of estimate and confidence interval. We applied progressively more stringent imputations.
Results: We found some examples robust to even the most stringent imputations (in which case we would not rate down for risk of bias); situations in which statistical significance was lost (if present), or observed (if absent), only for the most stringent assumptions (in which case one would rate down for risk of bias if one considered these stringent assumptions plausible); and situations in which statistical significance was lost if present, or observed if absent, in even less stringent imputations (in which case one would surely rate down for risk of bias). We observed instances in which application of our approach would lead to a decision to rate down for risk of bias when authors of the original systematic review concluded that missing data did not pose an important risk of bias problem.
Conclusion: This practical application of GRADE guidance documents the importance of the formal, structured evaluation of risk of bias due to missing data at the level of the meta-analysis.