Reasons for disagreements in 'Risk of bias' assessment for randomized controlled trials (RCTs) included in more than one Cochrane review

Tags: Oral
Bertizzolo L1, Bossuyt PM2, Atal I3, Ravaud P4, Dechartres A5
1INSERM, U1153 Epidemiology and Biostatistics Sorbonne Paris Cité Research Center (CRESS), Methods of therapeutic evaluation of chronic diseases Team (METHODS), Paris, F-75014 France; Paris Descartes University, Sorbonne Paris Cité, 2Department of Clinical Epidemiology, Biostatistics and Bioinformatics, Academic Medical Center, University of Amsterdam, 3Centre d'Épidémiologie Clinique, Hôpital Hôtel-Dieu, Paris, France; Team METHODS, INSERM U1153, Paris, France; Faculté de Médecine, Université Paris Descartes, Paris, 4Département d'Epidémiologie Clinique, APHP, Paris, France, Centre de recherche Inserm Epidémiologie et Statistique Paris Sorbonne Cité (U1153), Université Paris Descartes, Paris, 5Université Paris Descartes, Faculté de Médecine, INSERM U1153, Centre de Recherche Epidémiologie et Statistiques, Hôtel-Dieu (APHP), Paris

Background: The 'Risk of bias' (RoB) tool was developed by Cochrane to evaluate RoB in randomized controlled trials (RCTs). Despite detailed guidance and consensus agreement, concern was raised about its reproducibility. However, there is uncertainty about the main causes of disagreement.

Objectives: To assess reproducibility of RoB assessment in RCTs included in more than one Cochrane systematic review (SR); to evaluate whether disagreements were related to different information or different interpretation; to explore the main reasons for disagreement.

Methods: We obtained data from 2796 Cochrane SRs published between March 2011 and September 2014. We identified single RCTs included in different SRs and compared RoB assessment for the domains of random sequence generation (RSG), allocation concealment (AC), blinding of participants and personnel (BPP), blinding of outcome assessment (BOA), incomplete outcome data. For each domain, we calculated the proportion of agreement; two reviewers independently analyzed and classified disagreements as related to different information (e.g. some review authors had access to additional information) or different interpretation (same support for judgement but different interpretation); finally, we identified the main causes of disagreement.

Results: We identified 782 RCTs included in more than one SR with RoB assessment for the same item. The proportion of agreement was 82% for RSG (597/730), 74% for AC (581/781), 70% for BPP (143/205) and 70% for BOA (204/292). For AC, 68% of disagreements (136/200) were between low and unclear RoB. Disagreements were related to different interpretations in 72% of cases (145/200) and different information in 18% (37/200). The main reasons for different interpretation were more or less strict assessment of the same support for judgement (e.g. 'sealed envelopes' considered either at low, unclear, or high RoB) in 36% (52/145), and confusion with another RoB domain (e.g. confusion between AC and RSG) for 15% (22/145).

Conclusions: Our study shows that RoB assessment reproducibility remains suboptimal. Some disagreements are related to additional information obtained by some review authors. Other main causes of disagreement include more or less severe assessment, and confusion between domains.

Healthcare consumer involvement: Consumers have not been directly involved in the study.