Article type
Year
Abstract
Background: living guidelines have the potential to revolutionize the rapid translation of health research into policy and practice. Similar to other types of living evidence, living guidelines rely on the continual surveillance and integration of new evidence – a task potentially enabled by innovations, such as machine learning and citizen science. In the Stroke Living Guidelines Project we are exploring various approaches to develop an efficient evidence surveillance system.
Objectives: to evaluate the feasibility, accuracy and reliability of a surveillance system to identify evidence relevant to the Australian Stroke Foundation Clinical Guidelines for Stroke Management.
Methods: the Stroke Guidelines cover eight topic areas, comprising about 90 clinical questions. Using citations included in the 2017 edition of the guidelines, we developed broad-scope PubMed searches for systematic reviews, randomised trials, economic evaluations and studies of patient preferences and values. We transferred 11,700 records into Covidence covering May 2016 to December 2018 and set up monthly auto-alerts in PubMed from January 2019. Within Covidence, likely-relevant citations are being independently screened then allocated to topic areas for further assessment by members of the guideline working groups. We will then 1) compare the accuracy and efficiency of the manual search with the machine learning classifiers for RCTs, SRs and economic evaluations, and 2) assess the reliability of relying on PubMed alone by comparing retrieval from Epistemonikos, Cochrane CENTRAL and DORIS (Database Of Research In Stroke). In addition we are developing topic-based classifiers to semi-automate the triaging of records to the appropriate working groups.
Results: manual screening of over 70% of the backlog has been completed and over 500 records deemed relevant to the guideline questions. These records are now being triaged to the working groups for assessment and potential inclusion. The PubMed auto-alerts retrieve between 350 to 500 citations per month, of which 20% to 30% are duplicates (retrieved again following the addition of MeSH terms). Data on the accuracy and efficiency of machine classifiers, the reliability of restricting searches to PubMed, and the feasibility of semi-automated topic triage will be available by October 2019. We will present the effect of different approaches on the inclusion of studies in the guideline and possible impact on recommendations.
Conclusions: as living approaches to maintaining reviews and guidelines gain momentum, and machine learning tools become more routinely available, we need to continually evaluate the impact different approaches to evidence surveillance have on accuracy, efficiency, reliability and feasibility. The Stroke Living Guidelines Project is an exemplar of how to design and evaluate a living evidence surveillance system.
Patient or healthcare consumer involvement: no direct involvement
Objectives: to evaluate the feasibility, accuracy and reliability of a surveillance system to identify evidence relevant to the Australian Stroke Foundation Clinical Guidelines for Stroke Management.
Methods: the Stroke Guidelines cover eight topic areas, comprising about 90 clinical questions. Using citations included in the 2017 edition of the guidelines, we developed broad-scope PubMed searches for systematic reviews, randomised trials, economic evaluations and studies of patient preferences and values. We transferred 11,700 records into Covidence covering May 2016 to December 2018 and set up monthly auto-alerts in PubMed from January 2019. Within Covidence, likely-relevant citations are being independently screened then allocated to topic areas for further assessment by members of the guideline working groups. We will then 1) compare the accuracy and efficiency of the manual search with the machine learning classifiers for RCTs, SRs and economic evaluations, and 2) assess the reliability of relying on PubMed alone by comparing retrieval from Epistemonikos, Cochrane CENTRAL and DORIS (Database Of Research In Stroke). In addition we are developing topic-based classifiers to semi-automate the triaging of records to the appropriate working groups.
Results: manual screening of over 70% of the backlog has been completed and over 500 records deemed relevant to the guideline questions. These records are now being triaged to the working groups for assessment and potential inclusion. The PubMed auto-alerts retrieve between 350 to 500 citations per month, of which 20% to 30% are duplicates (retrieved again following the addition of MeSH terms). Data on the accuracy and efficiency of machine classifiers, the reliability of restricting searches to PubMed, and the feasibility of semi-automated topic triage will be available by October 2019. We will present the effect of different approaches on the inclusion of studies in the guideline and possible impact on recommendations.
Conclusions: as living approaches to maintaining reviews and guidelines gain momentum, and machine learning tools become more routinely available, we need to continually evaluate the impact different approaches to evidence surveillance have on accuracy, efficiency, reliability and feasibility. The Stroke Living Guidelines Project is an exemplar of how to design and evaluate a living evidence surveillance system.
Patient or healthcare consumer involvement: no direct involvement