Incorporation of assessments of risk of bias of primary studies in systematic reviews of randomized trials: a cross-sectional review

2013 Québec City

Hopewell S¹, Boutron I², Altman DG³, Ravaud P²

¹French Cochrane Centre, France; Centre for Statistics in Medicine, University of Oxford, UK; INSERM, U738, Paris, France; AP-HP (Assistance Publique des Hôpitaux de Paris), Hôpital Hôtel Dieu, Centre d’Epidémiologie Clinique, Paris, France; Univ. Paris Descartes, Sorbonne Paris Cité , Faculté de Médecine, Paris, France

²French Cochrane Centre, France; INSERM, U738, Paris, France; AP-HP (Assistance Publique des Hôpitaux de Paris), Hôpital Hôtel Dieu, Centre d’Epidémiologie Clinique, Paris, France; Univ. Paris Descartes, Sorbonne Paris Cité , Faculté de Médecine, Paris, France

³Centre for Statistics in Medicine, University of Oxford, UK; Statistics Methods Group, UK

Background:

Assessment of the validity of individual studies included in a systematic review, and the risk that they might overestimate the true intervention effect, is a critical part of the systematic review process.

Objective:

We examined how assessments of risk of bias (RoB) of primary studies are carried out and incorporated into the statistical analysis and overall findings of a systematic review.

Methods:

We assessed 200 systematic reviews of randomized trials published 2012; Cochrane (n = 100), non Cochrane (n = 100). Our primary outcome was a descriptive analysis of how assessments of RoB are carried out, methods used, and the extent to which such assessments were incorporated into the statistical analysis and review findings. We also evaluated differences between Cochrane and non Cochrane Reviews.

Results:

Most Cochrane Reviews (90%) assessed individual methodological components using the Cochrane RoB tool whereas non Cochrane Reviews (38%) were more likely to use a quality assessment scale; 20% of non Cochrane Reviews did not report the method used. Based on the assessment carried out by the authors of the systematic review, 58% (n = 116) of reviews had ≥ 1 trial at high RoB; median proportion of trials per review at high RoB was 50% (IQR 31–89%). Despite this only 56% (n = 65/116) incorporated the RoB assessment into the interpretation of the results in the abstract and 49% (n = 40/81) of Cochrane and 20% (n = 7/35) of non Cochrane Reviews incorporated the RoB assessment into the interpretation of the conclusions. Of the 166 (83%) systematic reviews which included a meta-analysis only 19 (11%) incorporated the RoB assessment into the statistical analysis.

Conclusion:

Cochrane Reviews were more likely than non Cochrane Reviews to report how RoB assessments of primary studies were carried out, however both frequently failed to take such assessments into account in the statistical analysis and conclusions of the systematic review.