Sensitivity, precision and efficiency of search strategies built using PubReMiner compared with current best practice search strategy building.

Oral
2024 Prague [Global Evidence Summit]
Dullea A1, Carrigan M2, O'Sullivan L3, Delaunois I4, Clark H5, Giusti M6, Walsh K7, Harrington P2, Smith S8, Ryan M9
1Discipline of Public Health & Primary Care, School of Medicine, Trinity College Dublin, The University of Dublin, Dublin, Ireland; Health Technology Assessment Directorate, Health Information and Quality Authority, Cork, Ireland
2Health Technology Assessment Directorate, Health Information and Quality Authority, Cork, Ireland
3Health Technology Assessment Directorate, Health Information and Quality Authority, Cork, Ireland; Health Research Board-Trials Methodology Research Network, College of Medicine, Nursing and Health Sciences, National University of Galway, , Galway, Ireland
4European Food Safety Authority, , Parma, Italy
5HSE Health Library Ireland, Sligo University Hospital, Sligo, Ireland
6Department of Experimental and Clinical Medicine, University of Florence, , Florence, Italy
7Health Technology Assessment Directorate, Health Information and Quality Authority, Cork, Ireland; School of Pharmacy, University College Cork, Cork, Ireland
8Discipline of Public Health & Primary Care, School of Medicine, Trinity College Dublin, The University of Dublin, Dublin, Ireland
9Health Technology Assessment Directorate, Health Information and Quality Authority, Cork, Ireland; Department of Pharmacology and Therapeutics, Trinity College Dublin, The University of Dublin, , Dublin, Ireland
Background
PubReMiner is a text-mining tool which analyses a seed set of citations to assess word frequency in titles, abstracts and medical subject headings to help build search strategies efficiently. This research is funded by the Health Research Board (Ireland) and the HSC Public Health Agency (Grant number ESI-2021-001) through Evidence Synthesis Ireland/Cochrane Ireland.
Objectives
This study aims to determine the sensitivity and precision of search strategies built using the PubReMiner tool.

Methods
This prospective comparative study embeds a methodological research question across multiple reviews and is registered as SWAR26 in the SWAR Repository Store. For each review, using the same research question, one librarian developed the conventional search strategy, while a second independently built a search strategy using PubReMiner. Both search strategies were developed by experienced librarians per work instructions, prepared and piloted a priori. Primary outcomes were sensitivity and precision. Secondary outcomes were number needed to read (NNR), number of unique references and efficiency (time for each librarian to construct search strategy).

Results
At time of submission of this abstract, preliminary analysis based on two searches conducted in 2023 (radioligand therapy for prostate cancer, screening for abdominal aortic aneurysm) have been conducted. Preliminary analysis found that the search strategies built using the PubReMiner tool found more unique records, but resulted in a higher NNR compared with those based on conventional search strategies. Our preliminary analysis indicated that strategies built using the PubReMiner approach were less sensitive and precise . This study is ongoing and will continue until 8-12 reviews are completed. Final analysis will be presented at the Global Evidence Summit 2024.

Conclusions
Results of this SWAR will be of interest to those considering using the PubReMiner tool, and to those investigating the use of supervised machine learning in search strategy building.
Statement on the relevance and importance:
It is anticipated that the results from this study will build on the evidence base for adopting text mining and AI tools in systematic reviews.