Article type
Year
Abstract
Background: The ACROBAT-NRSi tool is designed to promote rigour in the assessment of risk of bias (RoB) in non-randomised studies (NRS). The development and text of the ACROBAT-NRSi tool draws heavily on clinical examples to illustrate the tool and its application, but the feasibility of it being applied to public health evidence has not yet been fully explored.
Objectives: To assess the feasibility and usefulness of applying the ACROBAT-NRSi tool to NRS included in public health intervention reviews.
Methods: We applied the ACROBAT-NRSi tool to studies of housing improvement included in a published Cochrane Review. A group of five researchers, all with experience of critical appraisal, read the guidance and met to discuss queries and agree protocol level considerations before assessing the studies independently. Overall assessment of RoB was compared with two tools used to assess the studies in the review (Effective Public Health Practice (EPHP) tool and the Cochrane RoB tool).
Results: Agreement in assessments improved over time, but initially considerable discussion was required to agree on the interpretation of criteria and their application to complex social interventions. Specific issues included the incorporation of quasi-experimental study designs and the extent to which confounding may be uncontrollable.
The ACROBAT RoB for some studies was 'low' for two domains (intervention departure and measurement), but the overall assessment was 'serious' or 'critical' for all studies. Using the Cochrane RoB, all studies had two or more (range 2 to 8) domains assessed as 'high', implying a 'high' RoB overall. The EPHP overall assessment was 'low' for two studies, and 'moderate' for three studies.
Conclusions: ACROBAT-NRSi may benefit from further development to improve applicability to quasi-experimental studies of social interventions. The tool has limited sensitivity to identify variations in study quality. Where the RoB is 'serious' or 'critical' for a body of evidence, greater sensitivity to variations in study quality could allow differentiation of best available evidence from poorer quality evidence.
Objectives: To assess the feasibility and usefulness of applying the ACROBAT-NRSi tool to NRS included in public health intervention reviews.
Methods: We applied the ACROBAT-NRSi tool to studies of housing improvement included in a published Cochrane Review. A group of five researchers, all with experience of critical appraisal, read the guidance and met to discuss queries and agree protocol level considerations before assessing the studies independently. Overall assessment of RoB was compared with two tools used to assess the studies in the review (Effective Public Health Practice (EPHP) tool and the Cochrane RoB tool).
Results: Agreement in assessments improved over time, but initially considerable discussion was required to agree on the interpretation of criteria and their application to complex social interventions. Specific issues included the incorporation of quasi-experimental study designs and the extent to which confounding may be uncontrollable.
The ACROBAT RoB for some studies was 'low' for two domains (intervention departure and measurement), but the overall assessment was 'serious' or 'critical' for all studies. Using the Cochrane RoB, all studies had two or more (range 2 to 8) domains assessed as 'high', implying a 'high' RoB overall. The EPHP overall assessment was 'low' for two studies, and 'moderate' for three studies.
Conclusions: ACROBAT-NRSi may benefit from further development to improve applicability to quasi-experimental studies of social interventions. The tool has limited sensitivity to identify variations in study quality. Where the RoB is 'serious' or 'critical' for a body of evidence, greater sensitivity to variations in study quality could allow differentiation of best available evidence from poorer quality evidence.