Risk of Bias Assessment tool for Non-randomized Studies (RoBANS): Development and validation of a new instrument

Tags: Oral
Park J1, Lee Y1, Seo H2, Jang B1, Son H1, Kim S3, Shin S4, Hahn S5
1Department of Health Technology Assessment Research, National Evidence-based Healthcare Collaborating Agency, Seoul, Korea, 2Department of Nursing, College of Medicine, Chosun University, Gwangju, Korea, 3Department of Family Medicine, College of Medicine, Hallym University, Seoul, Korea, 4Department of Pulmonary Medicine, Ajou University, Suwon, Korea, 5Department of Medicine, College of Medicine, Seoul National University, Seoul, Korea

Background: Few validated checklist tools are available to assess the risk of bias (RoB) of non-randomized studies (NRS), although systematic reviews including NRS have increased in number, for many reasons.

Objectives: The aims of the study are to develop and validate a new RoB tool for NRS.

Methods: After deciding principles and reviewing relevant tools, we developed the Risk of Bias Assessment tool for Non-randomized Studies (RoBANS) tool. Subsequent testing involved the assessment of inter-rater agreement between reviewers, time to apply the RoBANS tool, degree of correlation for overall risk compared with overall quality scores, user-convenience, and content validity.

Results: The RoBANS contains 6 domains including the selection of participants, confounding variables, measurement of intervention (exposure), blinding of outcome assessment, incomplete outcome data and selective outcome reporting (table). Inter-rater agreement of the RoBANS tool varied in individual domains (weighted κ=0.38 to 0.74). The mean time to complete the RoBANS tool was less than for the MINORS (Methodological Items for NOn-Randomized Studies) (9.5 minutes (SD 3.39) per study vs. 10.45 (SD 3.54),( P = 0.18). There was a moderate correlation between overall RoB compared with the MINORS (Kendall’s τ=0.384). Reviewers evaluated RoBANS as 'good’ in terms of user-convenience for 3 items (mean 5.7 of 7 as the Likert scale; 0.81 %). External experts (n = 8) evaluated RoBANS as 'fair’ in terms of face validity for 7 items (mean 5.4 of 7 as the Likert scale) and all recommended its use to assess the RoB for NRS.

Conclusions: RoBANS is a valid tool designed to assess the RoB of NRS. Since RoBANS is harmonized with the Cochrane’s RoB tool and GRADE (Grading of Recommendations Assessment, Development and Evaluation), and can be incorporated into RevMan and GRADEpro, it appears to be useful to people undertaking systematic reviews.