Article type
Year
Abstract
Background: Due to the growing need to provide evidence syntheses at a faster pace, researchers have begun focusing on the exploration of rapid review methods. Rapid reviews often use only one reviewer for literature screening to accelerate the process. However, single-reviewer screening misses 13% of relevant studies, on average, compared to only 3% with dual-reviewer screening. Predictors of studies at high risk to be missed are unknown and guidance about methods to regain missed studies in the literature screening process of a review is lacking.
Objectives: The primary objective of this study is to identify predictors of incorrectly excluded studies in the abstract screening process. The secondary objective is to find methods to regain those missed studies.
Methods: We will use a database of 24,942 inclusion and exclusion decisions from a crowd of 280 reviewers at the abstract level, which was provided by Gartlehner et al. (2020). On average, reviewers screened each abstract 12 times. The database contains 80 references that are relevant at the full-text level. The abstract will be classified as “missed” if half or more reviewers incorrectly excluded it. Based on a subset of relevant studies that were missed during abstract screening, we will identify potential predictors (e.g. study design, sample size, impact factor of journal) and obtain a list. Potential predictor variables will be collected for each of the 80 relevant references independently by one investigator and checked by a second independent investigator. We will then perform regression analyses to test if possible predictor variables are associated with the risk of the study to be missed during abstract screening.
We will assume that study abstracts classified by over half of the reviewers as "includes" would be included in the full text review. We will use non-bibliographic database search methods (e.g. reference list checking, forward citation tracking, hand searching) to search for missed studies. All newly identified references through each non-bibliographic database source method will be cross-checked with missed studies. The investigators will record where and through which method the missed study was regained. We will rank search methods by their potential to regain missed studies.
Results: Final results will be available by the time of the Cochrane Colloquium.
Conclusions: The findings of this methods study could influence screening methods and point out which particular study characteristics should tip off reviewers to be particularly careful and attentive when reviewing. Additionally, this study will show whether there are methods available that can successfully retrieve those studies missed.
Patient or healthcare consumer involvement: No direct involvement in this methods study.
Objectives: The primary objective of this study is to identify predictors of incorrectly excluded studies in the abstract screening process. The secondary objective is to find methods to regain those missed studies.
Methods: We will use a database of 24,942 inclusion and exclusion decisions from a crowd of 280 reviewers at the abstract level, which was provided by Gartlehner et al. (2020). On average, reviewers screened each abstract 12 times. The database contains 80 references that are relevant at the full-text level. The abstract will be classified as “missed” if half or more reviewers incorrectly excluded it. Based on a subset of relevant studies that were missed during abstract screening, we will identify potential predictors (e.g. study design, sample size, impact factor of journal) and obtain a list. Potential predictor variables will be collected for each of the 80 relevant references independently by one investigator and checked by a second independent investigator. We will then perform regression analyses to test if possible predictor variables are associated with the risk of the study to be missed during abstract screening.
We will assume that study abstracts classified by over half of the reviewers as "includes" would be included in the full text review. We will use non-bibliographic database search methods (e.g. reference list checking, forward citation tracking, hand searching) to search for missed studies. All newly identified references through each non-bibliographic database source method will be cross-checked with missed studies. The investigators will record where and through which method the missed study was regained. We will rank search methods by their potential to regain missed studies.
Results: Final results will be available by the time of the Cochrane Colloquium.
Conclusions: The findings of this methods study could influence screening methods and point out which particular study characteristics should tip off reviewers to be particularly careful and attentive when reviewing. Additionally, this study will show whether there are methods available that can successfully retrieve those studies missed.
Patient or healthcare consumer involvement: No direct involvement in this methods study.