Article type
Abstract
Background:
Timely access to high-quality synthesis evidence is crucial for evidence-informed decision-making. Public health practitioners and decision-makers want answers to their questions fast. However, systematic reviews take time. Artificial intelligence (AI) provides an innovative approach to expedite reference screening while maintaining scientific rigor.
Objectives:
To demonstrate how the National Collaborating Centre for Methods and Tools (NCCMT) has leveraged AI for relevance screening into its Rapid Evidence Service (RES) and Health Evidence™ database management.
Methods:
Four AI features were integrated into the RES relevance screening process: (1) The reranking feature, used to prioritize the most relevant references to be screened; (2) a ranking report feature, which predicts the total number of included studies; (3) AI screening, which automatically screens studies based on prediction scores; and (4) a screening error report, which identifies studies that were potentially falsely excluded.
An AI preview and rank feature was used to support relevance screening for inclusion to Health Evidence™. A training set of references (n = 43,901) was used to train the AI and assign a score to each reference predicting the probability the reference is relevant. An optimal threshold was established to identify the greatest number of not relevant records with minimal errors. This threshold was retrospectively tested on a large set of studies (n = 89,932) previously screened by humans and subsequently integrated into the Health Evidence™ workflow.
Results:
The NCCMT’s RES has used AI for relevance for rapid reviews on various public health topics. For most questions, at least half of retrieved search results are automatically excluded through AI screening, thus saving substantial time in the manual title and abstract screening stage.
Between September 2020 and January 2024, the integration of AI technologies into the Health Evidence™ monthly workflow reduced the number of references needed to screen during this time from 413,158 to 85,109 (79.4% reduction). This is a savings of 656 staff hours.
Conclusions:
AI can reduce screening burden for evidence synthesis and database management. Adapting synthesis methods through the integration of AI can ensure public health practitioners and decision-makers have timely access to synthesized evidence to inform decision-making.
Timely access to high-quality synthesis evidence is crucial for evidence-informed decision-making. Public health practitioners and decision-makers want answers to their questions fast. However, systematic reviews take time. Artificial intelligence (AI) provides an innovative approach to expedite reference screening while maintaining scientific rigor.
Objectives:
To demonstrate how the National Collaborating Centre for Methods and Tools (NCCMT) has leveraged AI for relevance screening into its Rapid Evidence Service (RES) and Health Evidence™ database management.
Methods:
Four AI features were integrated into the RES relevance screening process: (1) The reranking feature, used to prioritize the most relevant references to be screened; (2) a ranking report feature, which predicts the total number of included studies; (3) AI screening, which automatically screens studies based on prediction scores; and (4) a screening error report, which identifies studies that were potentially falsely excluded.
An AI preview and rank feature was used to support relevance screening for inclusion to Health Evidence™. A training set of references (n = 43,901) was used to train the AI and assign a score to each reference predicting the probability the reference is relevant. An optimal threshold was established to identify the greatest number of not relevant records with minimal errors. This threshold was retrospectively tested on a large set of studies (n = 89,932) previously screened by humans and subsequently integrated into the Health Evidence™ workflow.
Results:
The NCCMT’s RES has used AI for relevance for rapid reviews on various public health topics. For most questions, at least half of retrieved search results are automatically excluded through AI screening, thus saving substantial time in the manual title and abstract screening stage.
Between September 2020 and January 2024, the integration of AI technologies into the Health Evidence™ monthly workflow reduced the number of references needed to screen during this time from 413,158 to 85,109 (79.4% reduction). This is a savings of 656 staff hours.
Conclusions:
AI can reduce screening burden for evidence synthesis and database management. Adapting synthesis methods through the integration of AI can ensure public health practitioners and decision-makers have timely access to synthesized evidence to inform decision-making.