Article type
Year
Abstract
Background:
Because of the growing need to provide evidence syntheses under time constraints, recent research has explored rapid review methods, which often employ single-reviewer literature screening. However, the single-reviewer screening process is error-prone; on average, 13% of relevant studies are missed. To date, it is unclear whether certain types of studies or publications have a higher risk of being falsely excluded than others.
Objectives:
The aim of our methods study with an observational design was to identify risk factors on a study-, abstract-, journal-, and reviewer-level for eligible studies to be falsely excluded during single-reviewer abstract screening.
Methods:
We used a database of 1,000 inclusion and exclusion decisions pertaining to 80 eligible references from a crowd-based randomized controlled trial assessing the accuracy of single-reviewer abstract screening. We gathered a list of potential risk factors for studies falsely excluded during single-reviewer abstract screening. One investigator collected potential risk factor variables independently, and a second investigator checked the data. We built a random forest model using R version 4.2.2 to identify variables that could predict the risk of false exclusion of studies. We split up the dataset into a training and a validation set. We trained the random forest classifier with the training set and used it to predict classes of “single false exclusion” for the validation set. We calculated Mean Decrease Accuracy and Mean Decrease Gini to rank the importance of the collected variables.
Results:
The random forest model ranked study design and the Scimago Journal & Country Rank (SJR) of the publishing journal as the variables with the highest importance for false exclusions. Cross-sectional studies, controlled-before-after studies, and publications in journals with an SJR ≤ 2.29 had a higher risk of being falsely excluded. Other variables, such as the screener being a native speaker, self-rated topic expertise, and previous review screening experience, were ranked as the least important. The model achieved a good area under the curve at 0.71.
Conclusions:
Our results indicate that study design and a low SJR of the journal may be important risk factors for studies to be falsely excluded during single-reviewer abstract screening. Investigators need to be cognizant of these risk factors when conducting reviews that include non-randomized evidence.
Patient or healthcare consumer involvement: Rapid reviews generally involve patients and healthcare providers in the review process in order to focus on patient-relevant health outcomes.
Because of the growing need to provide evidence syntheses under time constraints, recent research has explored rapid review methods, which often employ single-reviewer literature screening. However, the single-reviewer screening process is error-prone; on average, 13% of relevant studies are missed. To date, it is unclear whether certain types of studies or publications have a higher risk of being falsely excluded than others.
Objectives:
The aim of our methods study with an observational design was to identify risk factors on a study-, abstract-, journal-, and reviewer-level for eligible studies to be falsely excluded during single-reviewer abstract screening.
Methods:
We used a database of 1,000 inclusion and exclusion decisions pertaining to 80 eligible references from a crowd-based randomized controlled trial assessing the accuracy of single-reviewer abstract screening. We gathered a list of potential risk factors for studies falsely excluded during single-reviewer abstract screening. One investigator collected potential risk factor variables independently, and a second investigator checked the data. We built a random forest model using R version 4.2.2 to identify variables that could predict the risk of false exclusion of studies. We split up the dataset into a training and a validation set. We trained the random forest classifier with the training set and used it to predict classes of “single false exclusion” for the validation set. We calculated Mean Decrease Accuracy and Mean Decrease Gini to rank the importance of the collected variables.
Results:
The random forest model ranked study design and the Scimago Journal & Country Rank (SJR) of the publishing journal as the variables with the highest importance for false exclusions. Cross-sectional studies, controlled-before-after studies, and publications in journals with an SJR ≤ 2.29 had a higher risk of being falsely excluded. Other variables, such as the screener being a native speaker, self-rated topic expertise, and previous review screening experience, were ranked as the least important. The model achieved a good area under the curve at 0.71.
Conclusions:
Our results indicate that study design and a low SJR of the journal may be important risk factors for studies to be falsely excluded during single-reviewer abstract screening. Investigators need to be cognizant of these risk factors when conducting reviews that include non-randomized evidence.
Patient or healthcare consumer involvement: Rapid reviews generally involve patients and healthcare providers in the review process in order to focus on patient-relevant health outcomes.