Article type
Year
Abstract
Background:
The EMBASE project utilizes a crowd model to help screen search results for relevance within the Cochrane Central Register of Controlled Trials (CENTRAL). To help screeners, approximately 80 words or phrases were selected to be highlighted, some in yellow and some in red. The yellow highlights are considered more likely to appear in records eligible for CENTRAL. The red highlights are ‘warning’ highlights considered likely to appear in records to be rejected.
Objectives:
We sought to assess the usefulness of these highlighted words. Do they speed up the screening? What proportion of records screened in the first six months of the project: (1) contained only red highlights and were rejected?; (2) contained only yellow highlights and were accepted?; (3) does the location within the citation of the highlighted word indicate that a record is very likely a reject or not?; and (4) are certain highlights more effective than others?
Methods:
Our analyses were based on all records screened by the crowd from January to July 2014. A sample of EMBASE screeners were asked to complete a brief questionnaire regarding the usefulness of highlights. Another random sample was selected to screen a batch of records without the highlights activated. Their screening speed and accuracy was compared to those who screened with the highlights.
Results:
Complete analyses will be presented. Interim results indicate some key findings: screeners want and use the highlights a great deal; almost all accepted records contained yellow highlights; all records with red highlights in the title of the citation were rejects; records with yellow highlights found only in the final half of the abstract were almost always rejects.
Conclusions:
Highlighting words and phrases within search results plays a significant role in speeding up the assessment of citations. Screeners are overwhelmingly in favour of highlights, both yellow and red. Highlighted terms are designed to help guide the screener; they are not designed to negate the need for human intervention.
The EMBASE project utilizes a crowd model to help screen search results for relevance within the Cochrane Central Register of Controlled Trials (CENTRAL). To help screeners, approximately 80 words or phrases were selected to be highlighted, some in yellow and some in red. The yellow highlights are considered more likely to appear in records eligible for CENTRAL. The red highlights are ‘warning’ highlights considered likely to appear in records to be rejected.
Objectives:
We sought to assess the usefulness of these highlighted words. Do they speed up the screening? What proportion of records screened in the first six months of the project: (1) contained only red highlights and were rejected?; (2) contained only yellow highlights and were accepted?; (3) does the location within the citation of the highlighted word indicate that a record is very likely a reject or not?; and (4) are certain highlights more effective than others?
Methods:
Our analyses were based on all records screened by the crowd from January to July 2014. A sample of EMBASE screeners were asked to complete a brief questionnaire regarding the usefulness of highlights. Another random sample was selected to screen a batch of records without the highlights activated. Their screening speed and accuracy was compared to those who screened with the highlights.
Results:
Complete analyses will be presented. Interim results indicate some key findings: screeners want and use the highlights a great deal; almost all accepted records contained yellow highlights; all records with red highlights in the title of the citation were rejects; records with yellow highlights found only in the final half of the abstract were almost always rejects.
Conclusions:
Highlighting words and phrases within search results plays a significant role in speeding up the assessment of citations. Screeners are overwhelmingly in favour of highlights, both yellow and red. Highlighted terms are designed to help guide the screener; they are not designed to negate the need for human intervention.