Article type
Year
Abstract
Background: Although many studies validating clinical prediction rules are published, they have been unevenly focused on a few prediction rules leaving many without any external validation.
Objectives: This study aims to demonstrate research wastes related to conducting many external validation studies of a clinical prediction rule.
Methods: Data from published meta-analyses of Pneumonia Severity Index (PSI) and Alvarado Score were re-analyzed. From each validation study, the publication date, total number of subjects, and number of predicted and observed events were recorded. Random-effects cumulative meta-analyses of predictive performance (predicted/observed event ratio) were conducted according to the publication date. Then, the trajectory of previous to current cumulative predictive performance ratio over information step (addition of a new validation study) was graphically assessed. The number of validation studies and participants included in the validation studies were calculated before and after the stability of predictive performance is reached.
Results: Firstly, 30 validation studies of PSI which contained 26,563 participants were re-analyzed. After the data from the twelfth validation study was added to the recursive cumulative meta-analysis, the trajectory of cumulative predictive performance became stable (sustained less than 5% change). Therefore, 19 (63.3%) validation studies and the data from 17,443 (65.7%) participants added little value. Secondly, 34 studies validating Alvarado Score (9778 participants) were assessed. The trajectory of cumulative predictive performance became stable after the data from the seventh validation study was added to the recursive cumulative meta-analysis. Hence, 24 (80%) validation studies and data from 8066 (82.5%) participants included in these validations had little value. A recalibration was done in only 1 validation study of PSI.
Conclusions: Substantial research wastes were demonstrated in the validation of PSI and Alvarado Score. Before a validation of a clinical prediction rule is carried out, researchers should carefully consider whether it is truly necessary.
Objectives: This study aims to demonstrate research wastes related to conducting many external validation studies of a clinical prediction rule.
Methods: Data from published meta-analyses of Pneumonia Severity Index (PSI) and Alvarado Score were re-analyzed. From each validation study, the publication date, total number of subjects, and number of predicted and observed events were recorded. Random-effects cumulative meta-analyses of predictive performance (predicted/observed event ratio) were conducted according to the publication date. Then, the trajectory of previous to current cumulative predictive performance ratio over information step (addition of a new validation study) was graphically assessed. The number of validation studies and participants included in the validation studies were calculated before and after the stability of predictive performance is reached.
Results: Firstly, 30 validation studies of PSI which contained 26,563 participants were re-analyzed. After the data from the twelfth validation study was added to the recursive cumulative meta-analysis, the trajectory of cumulative predictive performance became stable (sustained less than 5% change). Therefore, 19 (63.3%) validation studies and the data from 17,443 (65.7%) participants added little value. Secondly, 34 studies validating Alvarado Score (9778 participants) were assessed. The trajectory of cumulative predictive performance became stable after the data from the seventh validation study was added to the recursive cumulative meta-analysis. Hence, 24 (80%) validation studies and data from 8066 (82.5%) participants included in these validations had little value. A recalibration was done in only 1 validation study of PSI.
Conclusions: Substantial research wastes were demonstrated in the validation of PSI and Alvarado Score. Before a validation of a clinical prediction rule is carried out, researchers should carefully consider whether it is truly necessary.