Background:

Hierarchical methods recommended (Leeflang et al 2008) for meta-analyses of diagnostic test accuracy studies are complex, relying on iterative procedures for the estimation of multiple model parameters. In certain circumstances, for instance when there are few studies in a meta-analysis, such models may not converge or produce unstable parameter estimates.

Objectives:

To evaluate the performance of meta-analytic approaches for test accuracy studies when few studies are available, and to develop recommendations for proceeding with meta-analysis when the suggested hierarchical methods fail.

Methods:

Ten-thousand meta-analysis datasets were simulated for each of a range of realistic scenarios that varied according comparisons of the accuracy of diagnostic tests to important factors, including: the number of studies, number
of patients within studies, disease prevalence, and heterogeneity in threshold and accuracy across studies. A variety of meta-analysis models were fitted and performance was assessed according to the bias, mean-square error and coverage of parameter estimates.

Results:

Irrespective of disease prevalence, estimation of hierarchical model parameters and their standard errors often fail in the absence of heterogeneity in threshold and accuracy. For example, in one scenario, the proportion of convergence failures for the hierarchical summary ROC model were 65%, 59% and 54% for 5, 10 and 15 studies respectively. Even given substantial heterogeneity, the models may still fail to converge when studies are few (e.g. 5 or less). Simpler models converge more easily but are clearly biased in a number of scenarios.

Conclusions:

Hierarchical methods are complex and their model convergence is sensitive to the number of studies in a meta-analysis and a variety of data characteristics. Where hierarchical methods fail, researchers may be forced to use simpler but less statistically rigorous methods, such as a univariate meta-analysis for sensitivity and specificity separately. Specific recommendations about how to proceed when hierarchical methods fail will be presented at the Colloquium.