Article type
Abstract
Background
The Cochrane Crowd microtasking platform started out offering a single task: identifying reports of randomised trials for Cochrane’s Central Register of Controlled Trials. A decade later, the platform has evolved significantly, attracting over 35,000 contributors from around the world who have contributed to 100s of tasks and amassed over 9 million individual classifications.
Methods
We conducted a range of evaluation studies of Cochrane Crowd which focussed on four key themes: accuracy of the crowd, efficiency in terms proportion of records that need resolving, engagement of the crowd, and the impact of crowdsourcing in wider evidence production workflows. This presentation will summarise the key findings from each of those studies, and detail potential future directions of the platform.
Results:
Evaluation of Cochrane Crowd’s agreement algorithms demonstrated very high collective accuracy of the crowd, ranging from 97-100% sensitivity across microtasks. Efficiency in terms of the proportion of records that did not need resolving due to discordant classifications ranged from 74-89%. Approximately 150 new people sign up to Cochrane Crowd every month with crowd contributors based in over 180 countries. The Cochrane Crowd initiative has helped to make CENTRAL the most comprehensive repository of randomised trials in the world; enabled rapid and accurate population of the Cochrane COVID-19 study register; helped to create training data for machine learning classifiers; and, via the Screen4Me service which uses Cochrane Crowd, helped to assess search results for over 200 Cochrane reviews.
Conclusion:
Cochrane Crowd was one of the first and is arguably the most successful implementation of crowdsourcing in evidence-based healthcare. It is a powerful example of generating economies of effort to produce results far greater than the sum of its parts. We will continue to develop this important human resource, understanding that people have always been our greatest asset. It has become a fundamental part of the evidence production ecosystem, and one that has the potential to transform the way we produce health evidence.
The Cochrane Crowd microtasking platform started out offering a single task: identifying reports of randomised trials for Cochrane’s Central Register of Controlled Trials. A decade later, the platform has evolved significantly, attracting over 35,000 contributors from around the world who have contributed to 100s of tasks and amassed over 9 million individual classifications.
Methods
We conducted a range of evaluation studies of Cochrane Crowd which focussed on four key themes: accuracy of the crowd, efficiency in terms proportion of records that need resolving, engagement of the crowd, and the impact of crowdsourcing in wider evidence production workflows. This presentation will summarise the key findings from each of those studies, and detail potential future directions of the platform.
Results:
Evaluation of Cochrane Crowd’s agreement algorithms demonstrated very high collective accuracy of the crowd, ranging from 97-100% sensitivity across microtasks. Efficiency in terms of the proportion of records that did not need resolving due to discordant classifications ranged from 74-89%. Approximately 150 new people sign up to Cochrane Crowd every month with crowd contributors based in over 180 countries. The Cochrane Crowd initiative has helped to make CENTRAL the most comprehensive repository of randomised trials in the world; enabled rapid and accurate population of the Cochrane COVID-19 study register; helped to create training data for machine learning classifiers; and, via the Screen4Me service which uses Cochrane Crowd, helped to assess search results for over 200 Cochrane reviews.
Conclusion:
Cochrane Crowd was one of the first and is arguably the most successful implementation of crowdsourcing in evidence-based healthcare. It is a powerful example of generating economies of effort to produce results far greater than the sum of its parts. We will continue to develop this important human resource, understanding that people have always been our greatest asset. It has become a fundamental part of the evidence production ecosystem, and one that has the potential to transform the way we produce health evidence.