'Other bias' in the 'Risk of bias' tool for Cochrane Reviews: a systematic analysis

Tags: Oral
Luo X1, Lv M2, Yu X3, Wang J3, Ma Y3, Liu X3, Estill J4, Yang K3, Chen Y3
1Evidence-Based Medicine Center of Lanzhou University, Chinese GRADE Center, WHO Collaborating Centre for Guideline Implementation and Knowledge Translation, School of Public Health, Lanzhou University, 2School of Public Health, Lanzhou University, 3Evidence-Based Medicine Center of Lanzhou University, Chinese GRADE Center, WHO Collaborating Centre for Guideline Implementation and Knowledge Translation, 4Institute of Global Health, University of Geneva, Institute of Mathematical Statistics and Actuarial Science, University of Bern

Background: the Cochrane 'Risk of bias' tool is frequently used for assessing the quality of Cochrane Reviews. So far no evaluation of 'other bias' (i.e. bias other than selection, performance, detection, attrition, or reporting) in Cochrane Reviews has been performed.

Objectives: to describe the risk of 'other bias' in Cochrane Reviews.

Methods: we conducted a survey of Cochrane Reviews of interventions, which were listed in the Cochrane Library in 2018. We abstracted data on the general characteristics of the reviews and the 'Other bias' in each randomized controlled trial (RCT) included in the reviews. Two review authors independently abstracted data from all eligible studies using a standardized form. We conducted descriptive data analyses including subgroup analyses by topic. We calculated the odds ratios for having low risk or unknown risk of other bias between the different subgroups.

Results: we identified a total of 611 eligible Cochrane Reviews involving 4882 RCTs. Of these, we identified 2187 (44.8%) as having low risk of other bias (e.g. not identified, not reported because the study was stopped early for benefit, none detected by the authors of the review, unlikely, no other known risks of bias, or baseline characteristics of the two groups being similar). We evaluated one-third of the RCTs (n = 1621; 33.2%) as having unclear risk (e.g. the trial was not registered, no detailed baseline characteristics, details of funding sources were not reported, no primary outcome defined, sample size calculation was not reported, study was supported by pharmaceutical company, or having no other known risks) and 649 (13.3%) RCTs as having high risk of other bias (e.g. lack of information for all risk of bias domains, no sample size calculations, insufficient information, contacting the corresponding author was unsuccessful, sponsored by pharmaceutical company, or low recruitment rate). The remaining 425 RCTs (8.7%) did not evaluate and report other bias. For low-risk evaluation, the topic of child health was higher than insurance medicine (odds ratio (OR) 3.47, P = 0.008) and neurology (OR = 4.19, P = 0.001). However, the unclear risk in child health fields was more common than the insurance medicine (OR = 2.46, P = 0.04) and insurance medicine (OR = 2.65, P = 0.009) in low-risk evaluation.

Conclusions: we judged only a few RCTs to have high risk of other bias; however in reality the risk may be greater. Cochrane Review authors should follow carefully the instructions in the Cochrane Handbook for Systematic Reviews of Interventions when conducting a Cochrane Review. In addition, the Cochrane Library should give more details and explanations of other bias in their Handbook to improve the quality and transparency of CRs.

Patient or healthcare consumer involvement: we hope this study could improve the reported rate of 'other bias' in risk of bias for the conduct of Cochrane Reviews.