Article type
Year
Abstract
Background: Although meta-analysis may synthesize estimates for a test’s accuracy, heterogeneity often blights these and it is not always clear whether clinicians should trust these estimates when applying them to their own practice.
Objectives: Apply a novel statistical method to determine whether the summary estimates of a test’s positive and negative predictive values (PPV and NPV) are likely to be valid in practice.
Methods: Using four test accuracy reviews as examples, a univariate meta-analysis model was used to derive estimates for the likelihood ratios. Based on previously reported methods this model was extended to produce a tailored estimate specific to the setting of interest. The PPV and NPV for the tests were the main outcomes of interest. A new validation statistic, Vn is introduced that assesses the validity of the standard and tailored estimate for the PPV and NPV in a new population. This is derived using a cross-validation procedure that compares the observed and expected post-test predictions in an omitted study on multiple occasions. A significant Vn (P value < 0.05) suggests the estimate would not be valid in a new population.
Results: For the four examples, each meta-analysis model derived four PPV estimates and four NPV estimates. Using the standard meta-analysis model, Vn was significant in seven of the eight estimates. In contrast, for tailored meta-analysis, Vn was significant in only three of the eight estimates suggesting that when a standard estimate is likely to be invalid a tailored estimate may be appropriate (Table 1). Furthermore the differences in estimates between the models could potentially affect patient management.
Conclusion: Statistical validation of a test’s PPVs and NPVs could be part of the synthesis process of a test evaluation meta-analysis. Both the likely validity of the standard and tailored estimate in a new population could be ascertained at this stage. In the examples analysed, standard meta-analysis seldom yielded a valid estimate for the test’s performance. Importantly, this was often remedied by taking a tailored approach to the meta-analysis, which was more likely to yield a valid estimate.
Objectives: Apply a novel statistical method to determine whether the summary estimates of a test’s positive and negative predictive values (PPV and NPV) are likely to be valid in practice.
Methods: Using four test accuracy reviews as examples, a univariate meta-analysis model was used to derive estimates for the likelihood ratios. Based on previously reported methods this model was extended to produce a tailored estimate specific to the setting of interest. The PPV and NPV for the tests were the main outcomes of interest. A new validation statistic, Vn is introduced that assesses the validity of the standard and tailored estimate for the PPV and NPV in a new population. This is derived using a cross-validation procedure that compares the observed and expected post-test predictions in an omitted study on multiple occasions. A significant Vn (P value < 0.05) suggests the estimate would not be valid in a new population.
Results: For the four examples, each meta-analysis model derived four PPV estimates and four NPV estimates. Using the standard meta-analysis model, Vn was significant in seven of the eight estimates. In contrast, for tailored meta-analysis, Vn was significant in only three of the eight estimates suggesting that when a standard estimate is likely to be invalid a tailored estimate may be appropriate (Table 1). Furthermore the differences in estimates between the models could potentially affect patient management.
Conclusion: Statistical validation of a test’s PPVs and NPVs could be part of the synthesis process of a test evaluation meta-analysis. Both the likely validity of the standard and tailored estimate in a new population could be ascertained at this stage. In the examples analysed, standard meta-analysis seldom yielded a valid estimate for the test’s performance. Importantly, this was often remedied by taking a tailored approach to the meta-analysis, which was more likely to yield a valid estimate.