A framework for developing and implementing clinical prediction models across multiple studies

Tags: Oral
Debray TPA1, Moons KGM1, Ahmed I2, Koffijberg H1, Riley RD2
1Julius Center for Health Sciences and Primary Care, The Netherlands, 2University of Birmingham, UK

Background: The availability of participant-level data from multiple sources is an increasingly prevalent phenomenon in prediction research. However, the corresponding populations typically differ in important aspects, such as baseline risk. This has driven the adoption of meta-analytical approaches for appropriately dealing with heterogeneity when combining such data. Unfortunately, these metaanalytical approaches do not provide a single prediction model that can readily be applied to new populations. Instead, they reveal the variability of baseline risk and predictor effects across studies, and provide little guidance about how then to proceed in integrating these findings.

Methods: We propose several approaches to account for heterogeneity in baseline risk and to obtain a valid model intercept when applying the model in a new population. We evaluate the consistency of model performance using the existing internal-external cross-validation approach, using an empirical Deep Vein Thrombosis dataset as an example.

Results: We found that the resulting prediction models are most generalizable when predictor effects are homogeneous. This can be achieved by excluding heterogeneous variables from the model or by including additional variables that explain heterogeneity of predictor effects. When baseline risks are heterogeneous, stratified estimation of the model intercept appears to be the best approach. An appropriate model intercept can then be derived from the outcome proportion in the population of interest, yielding superior calibration.

Conclusions: The approaches we propose can be used to develop a single, integrated prediction model from multiple IPDs that has superior generalizability. With minimal demographic information, the resulting model can be calibrated to new populations.