Article type
Year
Abstract
Background: In June 2014 the Cochrane Dementia and Cognitive Improvement Group began a programme of work to produce 12 intervention reviews looking at modifiable risk factors (MRFs) for dementia. To produce this large body of evidence the Cochrane Dementia Group have implemented a number of innovative methods in review production. This study focuses on the two innovative methods used in trial identification.
Objectives: Our objective was to identify all relevant trials for potential inclusion in 12 intervention reviews related to MRFs in dementia using innovative methods, including crowdsourcing, and to evaluate those methods against appropriate reference standards.
Methods: Two main innovations were utilised in the search and screen process. The first was to take a ‘suite’ approach and develop one search strategy per suite of related reviews rather than one search per review; and the second was to recruit a crowd through Students for Best Evidence to screen the search results using an online screening tool. Both methods were to be validated or compared against traditional methods.
Results: Taking a suite approach to the search meant that far fewer unique sets of search results were produced (four instead of 12). This meant that fewer overall results were identified with no compromise on the sensitivity of the searches. A total of 48 participants signed up to screen citations, with 23 (48%) screening 500 or more. A deadline of four weeks was given and met. Crowd performance, measured against random samples screened by members of the core teams (under evaluation), will be presented in terms of the crowds’ sensitivity (the collective ability to identify the RCTs correctly) and specificity (identify the rejects correctly).
Conclusions: We made gains in efficiency through having fewer citations to screen overall and by harnessing a crowd to screen those citations for trial design. This work contributes to the growing body of evidence on the beneficial role of crowdsourcing in the review production process. As we are confronted with ever increasing amounts of data to process, we need to find new methods to deal with the information overload.
Objectives: Our objective was to identify all relevant trials for potential inclusion in 12 intervention reviews related to MRFs in dementia using innovative methods, including crowdsourcing, and to evaluate those methods against appropriate reference standards.
Methods: Two main innovations were utilised in the search and screen process. The first was to take a ‘suite’ approach and develop one search strategy per suite of related reviews rather than one search per review; and the second was to recruit a crowd through Students for Best Evidence to screen the search results using an online screening tool. Both methods were to be validated or compared against traditional methods.
Results: Taking a suite approach to the search meant that far fewer unique sets of search results were produced (four instead of 12). This meant that fewer overall results were identified with no compromise on the sensitivity of the searches. A total of 48 participants signed up to screen citations, with 23 (48%) screening 500 or more. A deadline of four weeks was given and met. Crowd performance, measured against random samples screened by members of the core teams (under evaluation), will be presented in terms of the crowds’ sensitivity (the collective ability to identify the RCTs correctly) and specificity (identify the rejects correctly).
Conclusions: We made gains in efficiency through having fewer citations to screen overall and by harnessing a crowd to screen those citations for trial design. This work contributes to the growing body of evidence on the beneficial role of crowdsourcing in the review production process. As we are confronted with ever increasing amounts of data to process, we need to find new methods to deal with the information overload.