Article type
Abstract
Background: Guidance for performing systematic reviews requires reviewers to assess the risk of bias (RoB) of primary studies. The assessment of RoB of non-randomized studies of exposures requires unique considerations. A new tool, called ROBINS-E, has been proposed to assess the RoB of such studies.
Methods and Objectives: We applied the ROBINS-E tool in three systematic reviews, two of which addressed the health effects of nutritional exposures and one of which addressed drug harms, to 131 studies. A team comprising of three junior and three senior reviewers worked independently and in duplicate to assess the RoB of studies and resolved disagreements by discussion.
We describe our experiences applying the ROBINS-E tool, offering insights for future users and suggesting refinements.
Results: Some reviewers found the extensive guidance accompanying the ROBINS-E tool useful but others suggested that the length may reduce clarity and discourage reviewers from consulting the guidance.
Reviewers encountered challenges in making RoB judgments for studies addressing nutritional exposures due to insufficient guidance. For example, reviewers found that the tool did not provide sufficient guidance to make judgements regarding whether dietary measures are sufficiently valid and reliable.
Reviewers expressed concerns about the length of the tool and number of signalling questions, reporting an average application time of 44 minutes per study (SD: 12 minutes), which decreased with repeated use. For two systematic reviews, reviewers observed consistent ratings for select domains across all studies, suggesting the possibility of bypassing these domains to streamline RoB assessments.
We found several features of the ROBINS-E tool to be preferable to alternative RoB tools for non-randomized studies. For example, unlike previous tools, ROBINS-E does not conflate RoB with other characteristics of the evidence such as generalizability and precision, and accounts for RoB issues that are often neglected in other tools, such as selective reporting.
Conclusions: The ROBINS-E tool improves on previous RoB tools and integrates advancements in causal inference, such as assessing bias in relation to a target trial—a hypothetical trial that may or may not be feasible addressing the question of interest. Additional guidance, particularly for select health fields, may improve its application.
Methods and Objectives: We applied the ROBINS-E tool in three systematic reviews, two of which addressed the health effects of nutritional exposures and one of which addressed drug harms, to 131 studies. A team comprising of three junior and three senior reviewers worked independently and in duplicate to assess the RoB of studies and resolved disagreements by discussion.
We describe our experiences applying the ROBINS-E tool, offering insights for future users and suggesting refinements.
Results: Some reviewers found the extensive guidance accompanying the ROBINS-E tool useful but others suggested that the length may reduce clarity and discourage reviewers from consulting the guidance.
Reviewers encountered challenges in making RoB judgments for studies addressing nutritional exposures due to insufficient guidance. For example, reviewers found that the tool did not provide sufficient guidance to make judgements regarding whether dietary measures are sufficiently valid and reliable.
Reviewers expressed concerns about the length of the tool and number of signalling questions, reporting an average application time of 44 minutes per study (SD: 12 minutes), which decreased with repeated use. For two systematic reviews, reviewers observed consistent ratings for select domains across all studies, suggesting the possibility of bypassing these domains to streamline RoB assessments.
We found several features of the ROBINS-E tool to be preferable to alternative RoB tools for non-randomized studies. For example, unlike previous tools, ROBINS-E does not conflate RoB with other characteristics of the evidence such as generalizability and precision, and accounts for RoB issues that are often neglected in other tools, such as selective reporting.
Conclusions: The ROBINS-E tool improves on previous RoB tools and integrates advancements in causal inference, such as assessing bias in relation to a target trial—a hypothetical trial that may or may not be feasible addressing the question of interest. Additional guidance, particularly for select health fields, may improve its application.