Article type
Abstract
Background
Artificial Intelligence (AI) is increasingly used to support systematic reviews including semi-automate screening of titles and abstracts however, its use in full-text screening remains under-explored despite potential usefulness in expediting the process.
Objectives
To compare the accuracy and efficiency of full-text screening assisted by AI-generated suggestions and text highlights to conventional manual screening.
Methods
We selected a sample of 500 articles from a systematic review which inform The American Society of Hematology amyloidosis diagnostic guideline. We purposefully sampled articles that were found to be time-consuming and challenging (full papers rather than conference abstracts, studies with screening conflicts).
Two reviewers, independently and in-duplicate, initially screened the articles without the aid of automation (manual screening). The same articles were then subjected to a second round of screening by different team members than those that completed the manual screening, this time with the aid of AI-generated suggestions (AI-assisted screening). Figure 1 illustrates the study selection steps. The AI-model for suggestions was based on a set of pre-established inclusion/exclusion criteria. Figure 2 shows the Laser-AI tool interface using the AI-assisted screening.
To assess the similarity of the decisions, we compared the AI-assisted screening with the manual screening decisions, assumed to be the reference. We also compared the accuracy of decisions of AI-assisted single and double reviewer screening with manual screening. We then compared the time spent screening in both arms.
Results
After screening 58 references, we found that the decisions were similar between the two-reviewer AI-assisted screening and the manual screening (p=0.195). For single reviewer AI-assisted screening, similarity in decisions were found for reviewer 2 (p=0.223) but not reviewer 1 (p=0.005). Table 1 compares AI-assisted versus manual screening decisions.
Despite the AI-assisted screening only including full papers, a time saving of 7 seconds per study was found. Table 2 compares time spent for manual versus AI-assisted screening.
We plan to continue screening and present results based on a larger dataset during the conference.
Conclusions
Our preliminary results support that AI-assisted full-text screening could expedite the process of systematic reviews, allowing for more timely synthesis of results without compromising the quality of review.
Artificial Intelligence (AI) is increasingly used to support systematic reviews including semi-automate screening of titles and abstracts however, its use in full-text screening remains under-explored despite potential usefulness in expediting the process.
Objectives
To compare the accuracy and efficiency of full-text screening assisted by AI-generated suggestions and text highlights to conventional manual screening.
Methods
We selected a sample of 500 articles from a systematic review which inform The American Society of Hematology amyloidosis diagnostic guideline. We purposefully sampled articles that were found to be time-consuming and challenging (full papers rather than conference abstracts, studies with screening conflicts).
Two reviewers, independently and in-duplicate, initially screened the articles without the aid of automation (manual screening). The same articles were then subjected to a second round of screening by different team members than those that completed the manual screening, this time with the aid of AI-generated suggestions (AI-assisted screening). Figure 1 illustrates the study selection steps. The AI-model for suggestions was based on a set of pre-established inclusion/exclusion criteria. Figure 2 shows the Laser-AI tool interface using the AI-assisted screening.
To assess the similarity of the decisions, we compared the AI-assisted screening with the manual screening decisions, assumed to be the reference. We also compared the accuracy of decisions of AI-assisted single and double reviewer screening with manual screening. We then compared the time spent screening in both arms.
Results
After screening 58 references, we found that the decisions were similar between the two-reviewer AI-assisted screening and the manual screening (p=0.195). For single reviewer AI-assisted screening, similarity in decisions were found for reviewer 2 (p=0.223) but not reviewer 1 (p=0.005). Table 1 compares AI-assisted versus manual screening decisions.
Despite the AI-assisted screening only including full papers, a time saving of 7 seconds per study was found. Table 2 compares time spent for manual versus AI-assisted screening.
We plan to continue screening and present results based on a larger dataset during the conference.
Conclusions
Our preliminary results support that AI-assisted full-text screening could expedite the process of systematic reviews, allowing for more timely synthesis of results without compromising the quality of review.