Article type
Year
Abstract
Background: Knowledge users (KUs) need the highest-quality studies to make decisions about which interventions and policies should be used. The most reliable way to answer questions is with a systematic review (SR). AMSTAR and ROBIS tools are used to assess quality/bias in SRs. However, no automated tool currently exists to assess the quality/biases in SRs (Figure 1).
Objectives: (1) Develop a set of features to extract from SRs related to quality/bias; (2) develop a labelled dataset of 10,000 SRs; and (3) test, train and validate models and compare their accuracy.
Methods: A structure was proposed by the international steering group, and features were collected using a methods review and a survey. We used the results of a previous study comparing the tools which mapped an item’s concept across the three tools. A core team reviewed them and determined their feasibility in being automatically identifiable and predictable by our model (rather than through a manual process). A flowchart of activities is proposed (Figure 2). Five organisations with databases of preappraised SRs will supply the SRs. Duplicates will be removed and topics will be mapped. If topic fields, settings or conditions are missing, we will fill these gaps with a search for SRs. We will use ML Random Forests and DL Neural Network classification models such as Facebook’s StarSpace and Fasttext. We will use a ‘supervised’ learning model which learns by making predictions given examples of data.
Results: The proposed structure is found in Figure 1. The 21 items mapped from AMSTAR 1 and 2 and ROBIS will be used as the quality features. Other features chosen include PROGRESS-Plus items related to sex, gender and equity, 20 methods features (e.g., number of databases searched, PICO, and meta-analysis model used) and 10 results features (e.g., effect estimates and adverse events). SRs were collected and cleaned and the other features were extracted in duplicate. Testing will begin in March 2024.
Conclusions: An artificial intelligence tool that critically appraises SRs would dramatically reduce the financial and human resources currently needed to appraise SRs and update SR databases (e.g., McMasterPlus, HealthEvidence and SysVac).
Patient, public and/or healthcare consumer involvement: None.
Objectives: (1) Develop a set of features to extract from SRs related to quality/bias; (2) develop a labelled dataset of 10,000 SRs; and (3) test, train and validate models and compare their accuracy.
Methods: A structure was proposed by the international steering group, and features were collected using a methods review and a survey. We used the results of a previous study comparing the tools which mapped an item’s concept across the three tools. A core team reviewed them and determined their feasibility in being automatically identifiable and predictable by our model (rather than through a manual process). A flowchart of activities is proposed (Figure 2). Five organisations with databases of preappraised SRs will supply the SRs. Duplicates will be removed and topics will be mapped. If topic fields, settings or conditions are missing, we will fill these gaps with a search for SRs. We will use ML Random Forests and DL Neural Network classification models such as Facebook’s StarSpace and Fasttext. We will use a ‘supervised’ learning model which learns by making predictions given examples of data.
Results: The proposed structure is found in Figure 1. The 21 items mapped from AMSTAR 1 and 2 and ROBIS will be used as the quality features. Other features chosen include PROGRESS-Plus items related to sex, gender and equity, 20 methods features (e.g., number of databases searched, PICO, and meta-analysis model used) and 10 results features (e.g., effect estimates and adverse events). SRs were collected and cleaned and the other features were extracted in duplicate. Testing will begin in March 2024.
Conclusions: An artificial intelligence tool that critically appraises SRs would dramatically reduce the financial and human resources currently needed to appraise SRs and update SR databases (e.g., McMasterPlus, HealthEvidence and SysVac).
Patient, public and/or healthcare consumer involvement: None.