Article type
Abstract
Background: Crowdsourcing can support the conduct of systematic reviews. However, these approaches mainly encapsulate reviews that include one single type of study design. This project specifically focuses on engaging the public in screening of a dataset that combines 3 study designs: human, animal, and in vitro studies.
Objectives: To (1) pilot the feasibility of assembling a crowdsourcing project to engage citizens of the general public in a highly specialized screening task and (2) validate the training and screening outcomes of this approach. Two research questions were proposed: (1) the effects of ionizing radiation on the central nervous system (CNS) and (2) sex differences in exposure to ionizing radiation (SD).
Methods: Inclusion and exclusion criteria were formatted as a decision tree, mimicking the workflow of a field expert. Volunteers were trained to follow the decision tree. Five training calls were opened; access to the full screening dataset was granted to participants reaching recall >0.8 and specificity >0.4 on a training set of 25 prelabeled abstracts. Spanning the whole project, we built an active community with strong peer support and individualized feedback. The full screening phase is currently ongoing; the 3 groups are moderated in parallel (SD, CNS).
Results: In total, 1,300 citizens expressed their interest in contributing to the project. One hundred twenty participants passed the training stage and actively contribute to the project. From the active trainers, the most common error during the training stage was too low recall (= to harsh exclusion) as we aim at overinclusion at this stage. Intrarater decision-making agreed at >75% showing a stable approach; inter-rater decision-making varied between 50% and 100%. We adapted a new majority vote scheme with a minimum of 5 people agreeing at ≥75%.
Conclusion: (1) Long project timescales lead to loss of engagement by citizens and (2) scaling toward 3 study types and 51 countries requires a good data conduct combining trainings, stable platform access, and standardized data formats. We showed that a trained general crowd can make decisions comparable to domain experts, and it can be a useful tool to engage the public in conducting complex systematic reviews to address sustainable development goals.
Objectives: To (1) pilot the feasibility of assembling a crowdsourcing project to engage citizens of the general public in a highly specialized screening task and (2) validate the training and screening outcomes of this approach. Two research questions were proposed: (1) the effects of ionizing radiation on the central nervous system (CNS) and (2) sex differences in exposure to ionizing radiation (SD).
Methods: Inclusion and exclusion criteria were formatted as a decision tree, mimicking the workflow of a field expert. Volunteers were trained to follow the decision tree. Five training calls were opened; access to the full screening dataset was granted to participants reaching recall >0.8 and specificity >0.4 on a training set of 25 prelabeled abstracts. Spanning the whole project, we built an active community with strong peer support and individualized feedback. The full screening phase is currently ongoing; the 3 groups are moderated in parallel (SD, CNS).
Results: In total, 1,300 citizens expressed their interest in contributing to the project. One hundred twenty participants passed the training stage and actively contribute to the project. From the active trainers, the most common error during the training stage was too low recall (= to harsh exclusion) as we aim at overinclusion at this stage. Intrarater decision-making agreed at >75% showing a stable approach; inter-rater decision-making varied between 50% and 100%. We adapted a new majority vote scheme with a minimum of 5 people agreeing at ≥75%.
Conclusion: (1) Long project timescales lead to loss of engagement by citizens and (2) scaling toward 3 study types and 51 countries requires a good data conduct combining trainings, stable platform access, and standardized data formats. We showed that a trained general crowd can make decisions comparable to domain experts, and it can be a useful tool to engage the public in conducting complex systematic reviews to address sustainable development goals.