Article type
Abstract
Background
When conducting reviews of the effectiveness of complex interventions, often a variety of study types need to be included, resulting in the need for various critical appraisal tools. While GRADE takes into account different study types and their limitations, evidence profiles provide highly aggregated reasons for downgrading, which may be hard to understand for patients and decision-makers.
Objectives
Aggregating the results of different appraisal tools used in a real-world example
Methods
Based on a review of projects concerning patient safety, dimensions of critical appraisal tools for various study types were categorized into overarching domains, forming a matrix for overall bias assessment. When reporting results of the critical appraisal within the use case, studies raising concerns in any of the domains were counted.
Results
Within the use case, cohort studies (n = 3), cluster-randomized and stepped-wedge randomized trials (n = 4), and quasi-experimental studies (n = 4) were included. Domains covered by all corresponding critical appraisal tools (Cochrane’s RoB 2 for RCTs and the tools provided by the Joanna Briggs Institute for the other study types) were causal inference, which also covers randomization and confounding; composition and comparability of study groups at baseline; intervention fidelity; missing outcome data; reliability and validity of measurement tools; and inadequacy of statistical methods.
There were doubts about the potential causal inference in 4 of 11 studies, and 6 studies had the potential for bias due to composition and comparability of study groups. Within 2 studies, nonreliable and/or valid instruments for outcome measurement were applied. Missing outcome data and inadequate statistical methods were reported once.
Conclusions
Our approach aids the reporting of critical appraisal results for multiple study types and can serve as the basis for the overall assessment of certainty of evidence on multidimensional issues. It is still exemplary and needs further testing in evaluations of complex interventions and in combination with GRADE.
Relevance and importance to patients
Our approach helps to report sources of bias and, as such, downgrading in a transparent and overarching manner. Consequently, it may aid decision-makers and patients alike in assessing the true potential of complex interventions.
When conducting reviews of the effectiveness of complex interventions, often a variety of study types need to be included, resulting in the need for various critical appraisal tools. While GRADE takes into account different study types and their limitations, evidence profiles provide highly aggregated reasons for downgrading, which may be hard to understand for patients and decision-makers.
Objectives
Aggregating the results of different appraisal tools used in a real-world example
Methods
Based on a review of projects concerning patient safety, dimensions of critical appraisal tools for various study types were categorized into overarching domains, forming a matrix for overall bias assessment. When reporting results of the critical appraisal within the use case, studies raising concerns in any of the domains were counted.
Results
Within the use case, cohort studies (n = 3), cluster-randomized and stepped-wedge randomized trials (n = 4), and quasi-experimental studies (n = 4) were included. Domains covered by all corresponding critical appraisal tools (Cochrane’s RoB 2 for RCTs and the tools provided by the Joanna Briggs Institute for the other study types) were causal inference, which also covers randomization and confounding; composition and comparability of study groups at baseline; intervention fidelity; missing outcome data; reliability and validity of measurement tools; and inadequacy of statistical methods.
There were doubts about the potential causal inference in 4 of 11 studies, and 6 studies had the potential for bias due to composition and comparability of study groups. Within 2 studies, nonreliable and/or valid instruments for outcome measurement were applied. Missing outcome data and inadequate statistical methods were reported once.
Conclusions
Our approach aids the reporting of critical appraisal results for multiple study types and can serve as the basis for the overall assessment of certainty of evidence on multidimensional issues. It is still exemplary and needs further testing in evaluations of complex interventions and in combination with GRADE.
Relevance and importance to patients
Our approach helps to report sources of bias and, as such, downgrading in a transparent and overarching manner. Consequently, it may aid decision-makers and patients alike in assessing the true potential of complex interventions.