Article type
Year
Abstract
Background: Typical reviews deal with the 'information explosion’ by narrowing their search for studies (e.g., applying search filters). Relevant evidence can be missed through this approach. Current methods to minimise the risk of missing relevant studies involve searching broadly and screening potentially tens of thousands of records, which is not always practical. Resource-efficient approaches that maximise sensitivity are needed.
Objective: To evaluate whether new technologies allow us to search broadly without increasing the screening workload through semi-automated screening approaches. Specifically, we evaluate two types of text mining: a support vector machine using active learning (Wallace et al., 2010) and TerMine term clustering.
Methods: Text mining techniques were employed in an ongoing review to prioritise records for screening and to classify the records automatically as includes or excludes. Screening prioritisation was assessed by comparison with a 'baseline inclusion rate’ and through the novel application of power calculations. Classification was assessed through the stability of the classifier and the calculation of performance metrics (precision, recall, F-values). The classification procedure was also evaluated using simulations of completed reviews.
Results: Screening prioritisation worked when sufficient information was provided to the text mining tool; in the ongoing review, only 25% of all records were screened manually to identify the expected total number of included studies. Classification reduced the manual screening required in all reviews evaluated, although it worked better for some datasets than others.
Conclusions: Systematic reviews need to develop ways of handling the growing amount of evidence available. Text mining is a promising approach that shifts the emphasis of identification from the searching stage to screening. Reconceptualising searching permits broad searches to be conducted and allows reviewers to be more precise in estimating the number of potentially missing relevant studies than can be achieved by narrowing the search process. Areas for further development are suggested.
Objective: To evaluate whether new technologies allow us to search broadly without increasing the screening workload through semi-automated screening approaches. Specifically, we evaluate two types of text mining: a support vector machine using active learning (Wallace et al., 2010) and TerMine term clustering.
Methods: Text mining techniques were employed in an ongoing review to prioritise records for screening and to classify the records automatically as includes or excludes. Screening prioritisation was assessed by comparison with a 'baseline inclusion rate’ and through the novel application of power calculations. Classification was assessed through the stability of the classifier and the calculation of performance metrics (precision, recall, F-values). The classification procedure was also evaluated using simulations of completed reviews.
Results: Screening prioritisation worked when sufficient information was provided to the text mining tool; in the ongoing review, only 25% of all records were screened manually to identify the expected total number of included studies. Classification reduced the manual screening required in all reviews evaluated, although it worked better for some datasets than others.
Conclusions: Systematic reviews need to develop ways of handling the growing amount of evidence available. Text mining is a promising approach that shifts the emphasis of identification from the searching stage to screening. Reconceptualising searching permits broad searches to be conducted and allows reviewers to be more precise in estimating the number of potentially missing relevant studies than can be achieved by narrowing the search process. Areas for further development are suggested.