Article type
Abstract
"Background: In recent years, observational studies of routinely collected healthcare data (RCD) have become increasingly popular. Such studies rely on algorithms to identify health status of interest (e.g. diabetes or sepsis) for statistical analyses. However, the algorithm development and validation varied substantially, and their performances were often suboptimal which posed a serious threat to the study findings. However, these issues were often ignored.
Objectives: In response to these significant methodological gaps, we developed a guidance for the development, validation, and evaluation of algorithms for identifying health status (DEVELOP-RCD) by a systematic approach.
Methods: We undertook a narrative review and a systematic review of published studies that examined concepts and methodological issues about development, validation and evaluation of algorithms. Subsequently, we undertook an empirical study about an algorithm for identifying sepsis. Based on their findings, we generated the guidance containing a specific workflow and specific recommendations for algorithm development, validation and evaluation. Finally, a group of 20 external experts independently reviewed the guidance and held a consensus meeting to finalize the guidance.
Results: A standardized workflow regarding algorithm development, validation, and evaluation was developed. Guided by a targeted health status, the workflow consists of four cohesive steps — assessing an existing algorithm about its fitness for the target health status; developing a new algorithm with recommended methods; validating an algorithm by recommended performance measures; and evaluating the impact of algorithm on study results. Meanwhile, 13 good practice recommendations were established with elaborations. The practical study for identifying sepsis was also included to illustrate the use of the guidance.
Conclusion: A guidance is established to assist the researchers and clinicians for appropriately and accurately developing and applying algorithms to identify health status from RCD. This guidance may improve the trustworthiness of findings from observational studies of RCD.
"
Objectives: In response to these significant methodological gaps, we developed a guidance for the development, validation, and evaluation of algorithms for identifying health status (DEVELOP-RCD) by a systematic approach.
Methods: We undertook a narrative review and a systematic review of published studies that examined concepts and methodological issues about development, validation and evaluation of algorithms. Subsequently, we undertook an empirical study about an algorithm for identifying sepsis. Based on their findings, we generated the guidance containing a specific workflow and specific recommendations for algorithm development, validation and evaluation. Finally, a group of 20 external experts independently reviewed the guidance and held a consensus meeting to finalize the guidance.
Results: A standardized workflow regarding algorithm development, validation, and evaluation was developed. Guided by a targeted health status, the workflow consists of four cohesive steps — assessing an existing algorithm about its fitness for the target health status; developing a new algorithm with recommended methods; validating an algorithm by recommended performance measures; and evaluating the impact of algorithm on study results. Meanwhile, 13 good practice recommendations were established with elaborations. The practical study for identifying sepsis was also included to illustrate the use of the guidance.
Conclusion: A guidance is established to assist the researchers and clinicians for appropriately and accurately developing and applying algorithms to identify health status from RCD. This guidance may improve the trustworthiness of findings from observational studies of RCD.
"