Article type
Year
Abstract
Background: Publication bias and related biases that favor the inclusion of positive results in systematic reviews can lead to overly optimistic estimates of treatment benefit. Selection models offer one approach to adjusting meta-analyses for these biases. Several such models have been proposed, reviewed, and discussed in the literature, yet they are rarely applied, and little is known about their performance. The study reported here is the first systematic comparison of the performance of selection models to each other and to a funnel plot-based adjustment method, "trim and fill."
Objectives: To compare several selection models to each other and to "trim and fill."
Methods: Meta-analyses with a wide range of characteristics were simulated. We varied the number of studies (10 and 25), study sample size (20 to 200, 50 to 500, and 100 to 1500), mean odds ratio (OR) (0.5, 0.8, and 1.0), presence of heterogeneity (yes/no), and presence of publication bias (yes/no). The simulated probability of publication was a function of p-value and sample size. The selection models were step, half-normal, negative exponential, and logistic functions. These were used to model the probability of publication based on p-value alone, and on both p-value and sample size. We used maximum likelihood estimation, assuming random effects. Methods were compared on coverage probability (the probability that the nominal 95% confidence interval contained the true mean OR).
Results: For homogeneous meta-analyses with no publication bias, all methods had coverage probabilities close to 0.95. For homogeneous meta-analyses with publication bias, only the half-normal selection model had good coverage probabilities for all scenarios. For example, with the smallest meta-analyses (10 studies and sample size 20 to 200) and true odds ratio equal to 0.8, the coverage probabilities were 0.93, 0.92, 0.79, 0.80, 0.89, and 0.80 for half-normal, negative exponential, logistic, step function with two intervals, step function with three intervals, and "trim and fill", respectively. For heterogeneous meta-analyses with and without publication bias, none of the methods had consistently good coverage probabilities. However, the half-normal performed well when there was no publication bias, and the mean OR was 0.5 or 0.8, and when there was publication bias, and the mean OR was 0.5. The method used for incorporating sample size in the weight function did not improve coverage probability over the weight function based on p-value alone.
Conclusions: The selection model with half-normal weight function had good performance when adjusting the pooled estimate for homogeneous meta-analyses. The same conclusion held for heterogeneous meta-analyses, as long as the true mean treatment effect was large (OR=0.5). The appropriate inclusion of sample size in the weight function is a subject for future research. Another topic for future research is how to deal with a heterogeneity-induced relation between sample size and magnitude of treatment benefit. In such a case, selection models may perform poorly without inclusion of factors related to both sample size and treatment effect.
Objectives: To compare several selection models to each other and to "trim and fill."
Methods: Meta-analyses with a wide range of characteristics were simulated. We varied the number of studies (10 and 25), study sample size (20 to 200, 50 to 500, and 100 to 1500), mean odds ratio (OR) (0.5, 0.8, and 1.0), presence of heterogeneity (yes/no), and presence of publication bias (yes/no). The simulated probability of publication was a function of p-value and sample size. The selection models were step, half-normal, negative exponential, and logistic functions. These were used to model the probability of publication based on p-value alone, and on both p-value and sample size. We used maximum likelihood estimation, assuming random effects. Methods were compared on coverage probability (the probability that the nominal 95% confidence interval contained the true mean OR).
Results: For homogeneous meta-analyses with no publication bias, all methods had coverage probabilities close to 0.95. For homogeneous meta-analyses with publication bias, only the half-normal selection model had good coverage probabilities for all scenarios. For example, with the smallest meta-analyses (10 studies and sample size 20 to 200) and true odds ratio equal to 0.8, the coverage probabilities were 0.93, 0.92, 0.79, 0.80, 0.89, and 0.80 for half-normal, negative exponential, logistic, step function with two intervals, step function with three intervals, and "trim and fill", respectively. For heterogeneous meta-analyses with and without publication bias, none of the methods had consistently good coverage probabilities. However, the half-normal performed well when there was no publication bias, and the mean OR was 0.5 or 0.8, and when there was publication bias, and the mean OR was 0.5. The method used for incorporating sample size in the weight function did not improve coverage probability over the weight function based on p-value alone.
Conclusions: The selection model with half-normal weight function had good performance when adjusting the pooled estimate for homogeneous meta-analyses. The same conclusion held for heterogeneous meta-analyses, as long as the true mean treatment effect was large (OR=0.5). The appropriate inclusion of sample size in the weight function is a subject for future research. Another topic for future research is how to deal with a heterogeneity-induced relation between sample size and magnitude of treatment benefit. In such a case, selection models may perform poorly without inclusion of factors related to both sample size and treatment effect.