Article type
Year
Abstract
Background: Researchers within government, industry and non-profit research organizations increasingly employ systematic reviews to analyze and integrate the evidence available in peer-reviewed publications. A critical and time-consuming step in this process is screening the available literature to select relevant articles for further review.
Objectives: To evaluate the performance of SWIFT-ActiveScreener (ActiveScreener), a web-application that uses novel statistical and computational methods to prioritize articles for inclusion, while offering guidance on when additional screening will no longer yield additional relevant articles.
Methods: We tested ActiveScreener on 20 diverse systematic reviews for which human reviewers have previously screened more than 115,000 titles and abstracts.
Results: Compared to traditional screening, this method resulted in an average 54% reduction in screening burden, while still achieving 95% recall or higher; when tested on a subset of the 13 studies containing > 1000 articles, the reduction in screening burden improved to 71%. While these results are promising, machine-learning prioritization approaches can only be deployed confidently if users are sure that no relevant article will be missed in the process. Accordingly, ActiveScreener employs a novel algorithm to estimate recall while users work, thus providing a statistical basis for a decision about when to stop screening. Although this statistical confidence comes at a cost in terms of total number of articles screened, results indicate that, for large literature sets, the overall savings can still be large.
Conclusions:In SWIFT-ActiveScreener, these unique methodological advancements are implemented as a user-friendly application that allows users to manage their review, track its progress and provide conflict resolution. Together, these tools will enable researchers to perform literature screening faster, more cheaply and in a more reproducible manner.
Objectives: To evaluate the performance of SWIFT-ActiveScreener (ActiveScreener), a web-application that uses novel statistical and computational methods to prioritize articles for inclusion, while offering guidance on when additional screening will no longer yield additional relevant articles.
Methods: We tested ActiveScreener on 20 diverse systematic reviews for which human reviewers have previously screened more than 115,000 titles and abstracts.
Results: Compared to traditional screening, this method resulted in an average 54% reduction in screening burden, while still achieving 95% recall or higher; when tested on a subset of the 13 studies containing > 1000 articles, the reduction in screening burden improved to 71%. While these results are promising, machine-learning prioritization approaches can only be deployed confidently if users are sure that no relevant article will be missed in the process. Accordingly, ActiveScreener employs a novel algorithm to estimate recall while users work, thus providing a statistical basis for a decision about when to stop screening. Although this statistical confidence comes at a cost in terms of total number of articles screened, results indicate that, for large literature sets, the overall savings can still be large.
Conclusions:In SWIFT-ActiveScreener, these unique methodological advancements are implemented as a user-friendly application that allows users to manage their review, track its progress and provide conflict resolution. Together, these tools will enable researchers to perform literature screening faster, more cheaply and in a more reproducible manner.