Article type
Year
Abstract
Background:
Assessment of study risk of bias (RoB) is a key step in a systematic review but is very time-consuming. RobotReviewer is an open-access platform which partially automates RoB assessment using machine-learning (ML) and natural language processing. Covidence is a cloud-based systematic review production tool, recommended by Cochrane for the production of Cochrane Reviews.
Objectives:
The purpose of this trial is to evaluate the use of ML in RoB assessments in systematic reviews. Specifically, we sought to:
1) evaluate the accuracy of RoB assessments produced using a combination of human effort and ML compared to those produced by humans alone; and
2) determine whether the person-time required to complete RoB using a combination of human effort and ML is less than those completed by humans alone.
Methods:
A two-arm, parallel-group randomised controlled trial is ongoing. Each included study has initial RoB assessment performed by two independent reviewers (Human 1 and Human 2). RobotReviewer assistance will be provided to one of the two reviewers, with this assistance assigned randomly (1:1) for each study. RoB consensus will be performed by a third reviewer (Human 3) blinded to the allocation of RobotReviewer assistance.
Results:
The trial commenced recruitment in January, 2018 and is ongoing. At current rate of recruitment, final results will be available by mid-2018 and the co-primary outcomes will be presented.
Conclusions:
Following completion of the trial, results will be disseminated through journal publications and conference presentations. Study data will be used to inform next steps in machine-learning design and promotion.
Patient and healthcare consumer involvement:
While patients and consumers are unlikely to interact directly with the results of this study, increased automation holds exciting potential for increased availability of high quality, up-to-date health information. This trial is being conducted under the larger goal of building living evidence processes to provide reliable, up to date evidence for health decision-making.
Assessment of study risk of bias (RoB) is a key step in a systematic review but is very time-consuming. RobotReviewer is an open-access platform which partially automates RoB assessment using machine-learning (ML) and natural language processing. Covidence is a cloud-based systematic review production tool, recommended by Cochrane for the production of Cochrane Reviews.
Objectives:
The purpose of this trial is to evaluate the use of ML in RoB assessments in systematic reviews. Specifically, we sought to:
1) evaluate the accuracy of RoB assessments produced using a combination of human effort and ML compared to those produced by humans alone; and
2) determine whether the person-time required to complete RoB using a combination of human effort and ML is less than those completed by humans alone.
Methods:
A two-arm, parallel-group randomised controlled trial is ongoing. Each included study has initial RoB assessment performed by two independent reviewers (Human 1 and Human 2). RobotReviewer assistance will be provided to one of the two reviewers, with this assistance assigned randomly (1:1) for each study. RoB consensus will be performed by a third reviewer (Human 3) blinded to the allocation of RobotReviewer assistance.
Results:
The trial commenced recruitment in January, 2018 and is ongoing. At current rate of recruitment, final results will be available by mid-2018 and the co-primary outcomes will be presented.
Conclusions:
Following completion of the trial, results will be disseminated through journal publications and conference presentations. Study data will be used to inform next steps in machine-learning design and promotion.
Patient and healthcare consumer involvement:
While patients and consumers are unlikely to interact directly with the results of this study, increased automation holds exciting potential for increased availability of high quality, up-to-date health information. This trial is being conducted under the larger goal of building living evidence processes to provide reliable, up to date evidence for health decision-making.