Methodological considerations for assessing risk of bias in non-randomized studies

Tags: Oral
Sterne JA1, Reeves B1, Hernán M2, Savović J1, Turner L3, Higgins JP1, Development group for the Cochrane risk of bias tool for non-randomized studies .4
1University of Bristol, United Kingdom, 2Harvard School of Public Health, United States of America, 3Ottawa Hospital Research Institute, Canada, 4Various, International

Background: Cochrane intervention reviews rely primarily on evidence from randomized trials (RCTs), but this is not always available. Some clinical questions such as those about rare, long term or unexpected harms, are often addressed incompletely by RCTs. There is no widely accepted tool for assessing risk of bias in non-randomized studies (NRS) of interventions.

Objectives: Describe key methodological concepts that underpin development of the new Cochrane tool for assessing risk of bias in NRS.

Methods: We developed the tool through a wide consultation with methodologists, authors, Cochrane Review Groups and other stakeholders. Methodological considerations were addressed primarily through discussions within domain-focused working groups and a face-to-face meeting of all contributors.

Results: Evaluations of risk of bias in the results of NRS are facilitated by considering each study as an attempt to mimic a hypothetical RCT comparing the same interventions as the NRS (the ‘target trial’) and by specifying the intervention effect of interest (either assignment to the interventions at baseline, or receiving the interventions as specified in the protocol). Bias domains can be divided into “pre-intervention” domains (confounding and selection bias), an “at intervention” domain (bias in measuring interventions), and “post-intervention” domains (bias due to departures from intended interventions, missing data, measurement of outcomes and selection of the reported result). Assessment of pre-intervention bias domains is mainly distinct from assessments in RCTs (these focus on randomization procedures), but there is substantial overlap in assessment of post-intervention domains in RCTs and NRS. Studies that split follow-up according to intervention received require special consideration because confounding can occur at baseline and at the time of intervention change.

Conclusions: Evaluating risk of bias in a systematic review of NRS requires both methodological and content expertise. A structured approach to assessing risk of bias in different domains should provide a systematic way to organize and present the available evidence relating to risk of bias.