Article type
Year
Abstract
Background: In scoping reviews of broad evidence bases, boundaries of relevant evidence may be initially fuzzy, with a refined conceptual understanding of interventions and related phenomena of interest an intended output of the process rather than its starting point. Searches are therefore sensitive, retrieving large record sets that can be impractical to screen using conventional methods.
Objectives: To evaluate use of text-mining to reduce impractical screening workload in two large-scale scoping reviews of evidence for impacts of (i) choice architecture interventions (CA) and (ii) changes in the economic environment (EE) on health-related behaviours and corollary outcomes.
Methods: Baseline inclusion rates (BIRs) were estimated by screening random samples of records drawn from retrieved sets comprising over 800K (CA) and 1 million (EE) de-duplicated records. Text-mining technologies were applied to prioritise records for manual screening. 47 541 (CA) and 46 099 (EE) prioritised records were manually screened and observed inclusion rates (OIRs) recorded. Text-mining performance was measured in terms of OIRs relative to BIRs. Eligible records prioritised using text-mining were compared with those located using parallel snowball searches to assess unique yields and potential biases of each approach.
Results: Overall unadjusted OIRs were 10.1 (CA) and 8.3 (EE) times higher than BIRs. Text-mining reduced manual screening workload by 90% (CA) and 88% (EE) compared with conventional methods (absolute reductions of approximately 430 000 (CA) and 378 000 (EE) records), to identify 85% (CA) and 38% (EE) of remaining eligible records.
Conclusions: This study expands an emerging corpus of empirical evidence for use of text-mining to support screening, by demonstrating its feasibility, strengths and limitations in extremely large-scale scoping reviews. By reducing screening workload, text-miningmade it possible to assemble, describe and delimit large and complex evidence-bases that crossed research-disciplinary boundaries. Findings are transferable to other large-scale scoping and systematic reviews that incorporate conceptual or explanatory dimensions.
Objectives: To evaluate use of text-mining to reduce impractical screening workload in two large-scale scoping reviews of evidence for impacts of (i) choice architecture interventions (CA) and (ii) changes in the economic environment (EE) on health-related behaviours and corollary outcomes.
Methods: Baseline inclusion rates (BIRs) were estimated by screening random samples of records drawn from retrieved sets comprising over 800K (CA) and 1 million (EE) de-duplicated records. Text-mining technologies were applied to prioritise records for manual screening. 47 541 (CA) and 46 099 (EE) prioritised records were manually screened and observed inclusion rates (OIRs) recorded. Text-mining performance was measured in terms of OIRs relative to BIRs. Eligible records prioritised using text-mining were compared with those located using parallel snowball searches to assess unique yields and potential biases of each approach.
Results: Overall unadjusted OIRs were 10.1 (CA) and 8.3 (EE) times higher than BIRs. Text-mining reduced manual screening workload by 90% (CA) and 88% (EE) compared with conventional methods (absolute reductions of approximately 430 000 (CA) and 378 000 (EE) records), to identify 85% (CA) and 38% (EE) of remaining eligible records.
Conclusions: This study expands an emerging corpus of empirical evidence for use of text-mining to support screening, by demonstrating its feasibility, strengths and limitations in extremely large-scale scoping reviews. By reducing screening workload, text-miningmade it possible to assemble, describe and delimit large and complex evidence-bases that crossed research-disciplinary boundaries. Findings are transferable to other large-scale scoping and systematic reviews that incorporate conceptual or explanatory dimensions.