Article type
Year
Abstract
Background: systematic review is a formal process used widely in evidence-based toxicology and environmental health research to identify, assess, and integrate the primary scientific literature with the goal of answering a specific, targeted question in pursuit of the current scientific consensus. We recently received Phase I Small Business Innovation Research (SBIR) funding to conduct research and development to enhance our web-based, collaborative systematic review software application, SWIFT-Active Screener.
Objectives: by employing a machine learning methodology called “Active Learning”, and through a novel statistical method that can accurately estimate the percentage of relevant studies screened, Active Screener can significantly reduce the overall screening burden compared to traditional approaches.
Methods: we first investigated several improvements to our statistical algorithms used for article prioritization and recall estimation (Aim 1 – Improved Statistical Models). The resulting refinements further improve the performance of our algorithms and address critical technical issues that previously limited the applicability of our methods. Secondly, we explored ways in which our models and methods can be improved to handle the scenario in which an existing systematic review is updated with new data several years after its initial publication (Aim 2 – New Methods for Systematic Review Updates). Finally, in order to ensure that our software is capable of supporting the full demand from our many users, we have reengineered the system to support hundreds to thousands of simultaneous screeners (Aim 3 - Software Engineering for Scalability, Usability). During this research, our methods and software have been rigorously tested on 26 different systematic review datasets, demonstrating robust performance of Active Screener’s prioritization and recall estimation methods in a variety of real-world scenarios.
Results: for reviews with 5000 or more documents, we report an average reduction in screening burden of 61% (to obtain 95% recall). Active Screener has been used successfully to reduce the effort required to screen articles for systematic reviews conducted at a variety of organizations including NIEHS, EPA, USDA, TEDX, and EBTC. These early adopters have provided us with an abundance of useful data and user feedback, and we have identified several areas where we can continue to improve our methods and software. Several new features have been planned for the software, and it will be developed, improved and maintained for the foreseeable future.
Conclusions: SWIFT-Active Screener is a valuable tool that reduces the human screening burden and increases the pace at which reviews are conducted. On average, users can save in excess of 50% of screening effort normally required, resulting in significant time and cost savings.
Patient or healthcare consumer involvement: not applicable
Objectives: by employing a machine learning methodology called “Active Learning”, and through a novel statistical method that can accurately estimate the percentage of relevant studies screened, Active Screener can significantly reduce the overall screening burden compared to traditional approaches.
Methods: we first investigated several improvements to our statistical algorithms used for article prioritization and recall estimation (Aim 1 – Improved Statistical Models). The resulting refinements further improve the performance of our algorithms and address critical technical issues that previously limited the applicability of our methods. Secondly, we explored ways in which our models and methods can be improved to handle the scenario in which an existing systematic review is updated with new data several years after its initial publication (Aim 2 – New Methods for Systematic Review Updates). Finally, in order to ensure that our software is capable of supporting the full demand from our many users, we have reengineered the system to support hundreds to thousands of simultaneous screeners (Aim 3 - Software Engineering for Scalability, Usability). During this research, our methods and software have been rigorously tested on 26 different systematic review datasets, demonstrating robust performance of Active Screener’s prioritization and recall estimation methods in a variety of real-world scenarios.
Results: for reviews with 5000 or more documents, we report an average reduction in screening burden of 61% (to obtain 95% recall). Active Screener has been used successfully to reduce the effort required to screen articles for systematic reviews conducted at a variety of organizations including NIEHS, EPA, USDA, TEDX, and EBTC. These early adopters have provided us with an abundance of useful data and user feedback, and we have identified several areas where we can continue to improve our methods and software. Several new features have been planned for the software, and it will be developed, improved and maintained for the foreseeable future.
Conclusions: SWIFT-Active Screener is a valuable tool that reduces the human screening burden and increases the pace at which reviews are conducted. On average, users can save in excess of 50% of screening effort normally required, resulting in significant time and cost savings.
Patient or healthcare consumer involvement: not applicable