Article type
Year
Abstract
Background:
Recent research investigating methods used in systematic reviews (SRs) has found increasingly that critical appraisal of included studies is being performed. Critical appraisal constitutes an important input into the synthesis process as it allows reviewers to explore whether contradictory findings between studies reflect differences in methodologies, different risks of bias (RoB) or contextual factors. While some interventions (e.g. health system or policy interventions) are not easily amenable to randomised controlled trials (RCTs), there is a lack of agreement over the optimal approach to assessing quality for other study designs.
Objectives:
To examine if and how critical appraisals inform the synthesis and interpretation of evidence in SRs.
Methods:
All SRs published between March and May 2012 in 14 high ranked medical journals and a sample from The Cochrane Library were systematically assessed independently by two reviewers to determine if and how: critical appraisal was conducted; level of bias was summarised at study, domain and review levels; and study quality informed the synthesis process.
Results:
Of 59 SRs studied, all except six (90%) conducted critical appraisal of included studies, with most using or adapting a standardised tool. Many reviews conducted critical appraisal in a manner not allowing included studies to be ranked on the basis of quality. Assessments of study quality were not incorporated into the synthesis in a third (20) of the reviews. Common methods for incorporating quality into synthesis were sensitivity analysis, narrative assessment and exclusion of studies at high RoB. Nearly half of reviews investigating multiple outcomes that summarised bias at the study-level did not assess bias in relation to specific outcomes. Incorporating critical appraisal into synthesis occurred less often for reviews synthesising studies using non-RCT methods.
Conclusions:
The findings of SRs published in major journals are frequently uninformed by quality appraisal of included studies, even when critical appraisal has taken place. Further guidance is needed, particularly for SRs with a primary focus on non-RCTs.
Recent research investigating methods used in systematic reviews (SRs) has found increasingly that critical appraisal of included studies is being performed. Critical appraisal constitutes an important input into the synthesis process as it allows reviewers to explore whether contradictory findings between studies reflect differences in methodologies, different risks of bias (RoB) or contextual factors. While some interventions (e.g. health system or policy interventions) are not easily amenable to randomised controlled trials (RCTs), there is a lack of agreement over the optimal approach to assessing quality for other study designs.
Objectives:
To examine if and how critical appraisals inform the synthesis and interpretation of evidence in SRs.
Methods:
All SRs published between March and May 2012 in 14 high ranked medical journals and a sample from The Cochrane Library were systematically assessed independently by two reviewers to determine if and how: critical appraisal was conducted; level of bias was summarised at study, domain and review levels; and study quality informed the synthesis process.
Results:
Of 59 SRs studied, all except six (90%) conducted critical appraisal of included studies, with most using or adapting a standardised tool. Many reviews conducted critical appraisal in a manner not allowing included studies to be ranked on the basis of quality. Assessments of study quality were not incorporated into the synthesis in a third (20) of the reviews. Common methods for incorporating quality into synthesis were sensitivity analysis, narrative assessment and exclusion of studies at high RoB. Nearly half of reviews investigating multiple outcomes that summarised bias at the study-level did not assess bias in relation to specific outcomes. Incorporating critical appraisal into synthesis occurred less often for reviews synthesising studies using non-RCT methods.
Conclusions:
The findings of SRs published in major journals are frequently uninformed by quality appraisal of included studies, even when critical appraisal has taken place. Further guidance is needed, particularly for SRs with a primary focus on non-RCTs.