Article type
Abstract
Background: Automation methods, including artificial intelligence, offer potential efficiency gains in systematic reviews (SRs) and meta-analyses (MAs). Understanding their current usage is crucial for advancing evidence synthesis methodology.
Objectives: This study aims to investigate the prevalence and patterns of automation tool utilization across various stages of SRs and MAs.
Methods: A cross-sectional metaresearch survey was conducted, examining SRs and MAs published in August 2023 in Medline (via Ovid). We applied a restriction on date and selected the first fully-indexed month of 2023 based on the start date of searches for this study (08.02.2024). We included any type of review answering any research question. In the first part, we analyzed a subsample of protocols published in journals of SRs and MAs and extracted information about type of planned review, applied automation tools, review phases with automation, adhering to reporting tools. In the second part, we plan to analyze remaining full texts using Elicit tool to automatize the extraction process.
Results: After deduplication, we screened in pairs 4402 records and included 3320 for full text assessment. Within this sample, we identified 71 protocols. Conflicts were resolved through discussion. Preliminary analysis reveals diverse automation tool usage, primarily observed in the screening, data extraction and data analysis phases. Multiple tools such as Endnote, Rayyan, Covidence and GRADE Pro are employed, with variations in reporting standards (e.g. PRISMA checklist) across different review types and journals. Despite reporting on using specific software, the authors do not provide sufficient details on the phase of SRs or MAs to which it was applied. Almost none of the included protocols explicitly indicated the automation process, and few studies only indicated the use of artificial intelligence methods.
Conclusions: Automation methods are increasingly incorporated into SRs and MAs. However, standardization in reporting practices and tool selection is warranted to ensure transparency and reproducibility. We think this study has the potential to inform about automation tools and its prevalence within SRs and MAs. The findings may contribute to developing guidance to accelerate the process of conducting SRs and MAs with less time and resource wasting.
Objectives: This study aims to investigate the prevalence and patterns of automation tool utilization across various stages of SRs and MAs.
Methods: A cross-sectional metaresearch survey was conducted, examining SRs and MAs published in August 2023 in Medline (via Ovid). We applied a restriction on date and selected the first fully-indexed month of 2023 based on the start date of searches for this study (08.02.2024). We included any type of review answering any research question. In the first part, we analyzed a subsample of protocols published in journals of SRs and MAs and extracted information about type of planned review, applied automation tools, review phases with automation, adhering to reporting tools. In the second part, we plan to analyze remaining full texts using Elicit tool to automatize the extraction process.
Results: After deduplication, we screened in pairs 4402 records and included 3320 for full text assessment. Within this sample, we identified 71 protocols. Conflicts were resolved through discussion. Preliminary analysis reveals diverse automation tool usage, primarily observed in the screening, data extraction and data analysis phases. Multiple tools such as Endnote, Rayyan, Covidence and GRADE Pro are employed, with variations in reporting standards (e.g. PRISMA checklist) across different review types and journals. Despite reporting on using specific software, the authors do not provide sufficient details on the phase of SRs or MAs to which it was applied. Almost none of the included protocols explicitly indicated the automation process, and few studies only indicated the use of artificial intelligence methods.
Conclusions: Automation methods are increasingly incorporated into SRs and MAs. However, standardization in reporting practices and tool selection is warranted to ensure transparency and reproducibility. We think this study has the potential to inform about automation tools and its prevalence within SRs and MAs. The findings may contribute to developing guidance to accelerate the process of conducting SRs and MAs with less time and resource wasting.