Article type
Year
Abstract
Background: The assessment of risk of bias (RoB) in systematic reviews has recently evolved with the development of a Cochrane RoB tool for non-randomised studies of interventions (ACROBAT-NRSI). The tool evaluates internal validity of non-randomised studies (NRS) through signalling questions addressing domains of risk of bias. One of the domains assesses bias in measurement of outcomes, with no distinction between efficacy or harmful outcomes.
Objectives: To investigate potential bias with regard to a set of signalling questions proposed to investigate misclassification of adverse outcomes.
Methods: We used a sample of NRS from a systematic review of a specific adverse effect (lipodystrophy induced by antiretroviral therapy). The signalling questions included: 1) use of precise definition of the adverse effect, 2) active mode of harms data collection, 3) accurate mode of harms detection, 4) comparability of outcome assessment across intervention groups, 5) specification of who ascertained the adverse effect, 6) training of the person who ascertained the adverse effect, and 7) reporting of timing and frequency of harms data collection.
Results: Despite the fact that all studies (n = 20) were designed to investigate the adverse effect lipodystrophy, 25% failed to provide a clear definition for the outcome. Studies did not specify either who ascertained the adverse effect nor the training level of the staff (20% and 25%, respectively), were not designed to actively and accurately detect the adverse effect (15% and 20%), and failed in reporting timing and frequency of harms data collection (15%). Comparability of outcome assessment across intervention groups was unclear in 10% of the studies. In summary, 45% of the NRS were considered at moderate, serious or critical risk of bias.
Conclusions: A significant flaw in the assessment of adverse outcomes was found in a sample of NRS designed to investigate a pre-specified adverse effect. It is expected that a substantial error in the detection of harm outcomes may also be found in intervention trials. Further research on the impact of differential measurement error on the detection of adverse outcomes is required.
Objectives: To investigate potential bias with regard to a set of signalling questions proposed to investigate misclassification of adverse outcomes.
Methods: We used a sample of NRS from a systematic review of a specific adverse effect (lipodystrophy induced by antiretroviral therapy). The signalling questions included: 1) use of precise definition of the adverse effect, 2) active mode of harms data collection, 3) accurate mode of harms detection, 4) comparability of outcome assessment across intervention groups, 5) specification of who ascertained the adverse effect, 6) training of the person who ascertained the adverse effect, and 7) reporting of timing and frequency of harms data collection.
Results: Despite the fact that all studies (n = 20) were designed to investigate the adverse effect lipodystrophy, 25% failed to provide a clear definition for the outcome. Studies did not specify either who ascertained the adverse effect nor the training level of the staff (20% and 25%, respectively), were not designed to actively and accurately detect the adverse effect (15% and 20%), and failed in reporting timing and frequency of harms data collection (15%). Comparability of outcome assessment across intervention groups was unclear in 10% of the studies. In summary, 45% of the NRS were considered at moderate, serious or critical risk of bias.
Conclusions: A significant flaw in the assessment of adverse outcomes was found in a sample of NRS designed to investigate a pre-specified adverse effect. It is expected that a substantial error in the detection of harm outcomes may also be found in intervention trials. Further research on the impact of differential measurement error on the detection of adverse outcomes is required.