Article type
Year
Abstract
(Please note Part 2)
Background: a crucial step in any systematic review is the screening of abstracts. To reduce the risk of excluding relevant studies, international methodological guidance recommends that at least two investigators independently screen each abstract. By contrast, the Cochrane Handbook of Systematic Reviews of Interventions 6.0 explicitly states that it is “acceptable that initial screening of titles and abstracts is undertaken by one person”.
Rapid reviews, an increasingly popular form of evidence synthesis, often employ single-reviewer abstract screening to expedite review production. To date, the evidence base regarding the magnitude of bias that single-reviewer abstract screening introduces, however, is scarce. Published studies indicated that single-reviewer screening, on average, missed 8% of eligible studies (range 0% to 24%). A substantial limitation of these studies, however, is that results were highly dependent on the screening skills and experience of individual review authors.
Objectives: to assess the accuracy of single- compared with dual-reviewer abstract screening using a crowd-based approach.
Methods: we are assessing the accuracy of single-reviewer abstract screening using a diagnostic framework approach. Two published systematic reviews, one on a pharmacological topic, one on a public health topic serve as reference standards. We use the Cochrane Crowd platform and its crowdsourcing capabilities to create a large enough reviewer base (more than 250) to eliminate the impact of subjective skills and expertise of individual screeners. Cochrane Crowd randomly assigns participants, who have successfully completed a training exercise, to 100 randomly selected abstracts. At least 10 participants will screen each abstract. The reference standards are the included or excluded studies of the two systematic reviews. The primary outcome is the proportion of correct decisions of single screeners compared to the decisions of two screeners in the reference standards. We will calculate the average proportion of correct and incorrect decisions (i.e. sensitivities and specificities) over all raters with the corresponding 95% confidence intervals.
Results: the study is ongoing. Results will be available for the Cochrane Colloquium.
Conclusions: we cannot draw conclusions yet. Results will provide important information about the trade-off in validity that the more rapid single-reviewer screening has compared with the more time-intensive dual-reviewer screening.
Patient or healthcare consumer involvement: rapid reviews generally involve patients and healthcare providers in the review process in order to focus on patient-relevant health outcomes. For well-informed decision making, rapid reviews present the synthesized evidence conclusively in a simple and concise way.
Background: a crucial step in any systematic review is the screening of abstracts. To reduce the risk of excluding relevant studies, international methodological guidance recommends that at least two investigators independently screen each abstract. By contrast, the Cochrane Handbook of Systematic Reviews of Interventions 6.0 explicitly states that it is “acceptable that initial screening of titles and abstracts is undertaken by one person”.
Rapid reviews, an increasingly popular form of evidence synthesis, often employ single-reviewer abstract screening to expedite review production. To date, the evidence base regarding the magnitude of bias that single-reviewer abstract screening introduces, however, is scarce. Published studies indicated that single-reviewer screening, on average, missed 8% of eligible studies (range 0% to 24%). A substantial limitation of these studies, however, is that results were highly dependent on the screening skills and experience of individual review authors.
Objectives: to assess the accuracy of single- compared with dual-reviewer abstract screening using a crowd-based approach.
Methods: we are assessing the accuracy of single-reviewer abstract screening using a diagnostic framework approach. Two published systematic reviews, one on a pharmacological topic, one on a public health topic serve as reference standards. We use the Cochrane Crowd platform and its crowdsourcing capabilities to create a large enough reviewer base (more than 250) to eliminate the impact of subjective skills and expertise of individual screeners. Cochrane Crowd randomly assigns participants, who have successfully completed a training exercise, to 100 randomly selected abstracts. At least 10 participants will screen each abstract. The reference standards are the included or excluded studies of the two systematic reviews. The primary outcome is the proportion of correct decisions of single screeners compared to the decisions of two screeners in the reference standards. We will calculate the average proportion of correct and incorrect decisions (i.e. sensitivities and specificities) over all raters with the corresponding 95% confidence intervals.
Results: the study is ongoing. Results will be available for the Cochrane Colloquium.
Conclusions: we cannot draw conclusions yet. Results will provide important information about the trade-off in validity that the more rapid single-reviewer screening has compared with the more time-intensive dual-reviewer screening.
Patient or healthcare consumer involvement: rapid reviews generally involve patients and healthcare providers in the review process in order to focus on patient-relevant health outcomes. For well-informed decision making, rapid reviews present the synthesized evidence conclusively in a simple and concise way.