Impact of the use of Artificial Intelligence on conclusions of systematic reviews

2024 Prague [Global Evidence Summit]

Zaror C¹, Ávila-Oliver C², Figueiredo N³, Verdugo-Paiva F⁴, Troncoso-Gajardo P⁵, Rada G⁶, Brignardello-Petersen R⁷

¹Center for Research in Epidemiology, Economics and Oral Public Health (CIEESPO), Faculty of Dentistry, Universidad de La Frontera, Temuco, Chile; Department of Health Research Methods, Evidence, and Impact, McMaster University, Hamilton, Ontario, Canada

² Epistemonikos Foundation, Santiago, Chile; Escuela de Odontología, Facultad de Odontología y Ciencias de la Rehabilitación, Universidad San Sebastián, Santiago, Chile

³Center for Research in Epidemiology, Economics and Oral Public Health (CIEESPO), Faculty of Dentistry, Universidad de La Frontera, Temuco, Chile

⁴ Epistemonikos Foundation, Santiago, Chile; Department of Paediatrics, Obstetrics & Gynaecology and Preventive Medicine and Public Health at the Universitat Autònoma de Barcelona, Barcelona, España

⁵Departamento de Pediatría Estomatológica, Facultad de Odontología, Universidad de Talca, Talca, Chile

⁶ Epistemonikos Foundation, Santiago, Chile

⁷Department of Health Research Methods, Evidence, and Impact, McMaster University, Hamilton, Ontario, Canada

Background:

Rapid reviews allow producing evidence for stakeholders timely. Therefore, review teams should seek mechanisms to streamline the methods without impacting the review's conclusions. The Sustainable Knowledge Platform is a new system based on artificial intelligence(AI) to search and screen studies in the context of a systematic review process.

Objectives:

To evaluate the impact of AI versus traditional approach on effect estimates from meta-analyses.

Methods:

We conducted a meta-epidemiological study using data abstracted from randomized trials included in a dental network meta-analysis(NMA) about the prevention of occlusal caries in primary teeth. For the traditional approach, we searched MEDLINE, CENTRAL, EMBASE, CINAHL and LILACS databases through January 2024. Two researchers independently reviewed articles identified in the literature searches by examining them in the three consecutive phases of titles, abstracts and full-text. For the AI approach, we will match the components of the review question to the relevant terms of the taxonomy, run the predeveloped boolean strategies, and apply the automated classifiers to identify articles to include. Then, two researchers will carry out the data extraction of the articles included in both approaches. Subsequently, we will conduct two pairwise meta-analyses using a frequentist approach and a random effects model using data extracted from articles identified through the traditional approach (reference standard) and AI approach. To determine if there are important differences between the results of the two analyses, for each outcome in each comparison, we will use the threshold above which the NMA considered the effect was important (figure 1). Finally, we will calculate the proportion of outcomes for which the results are importantly different when using the data abstracted identified by AI versus the traditional approach. In addition, we will calculate the error rate of studies identified by the AI approach.

Results:

This study is ongoing, and the results will be presented at Global Evidence Submit as available.

Conclusion:

We think that effect estimates using data abstraction from the AI approach are not importantly different than those from the traditional approach, despite the difference of the studies included with both methods. AI could contribute to providing timely evidence to benefit patients.