Article type
Year
Abstract
Objective: To investigate two statistical tests for the detection of bias in meta-analysis (Egger et al., 1997; Begg and Mazumdar, 1994) with binary outcomes in a simulation study.
Methods: Binary outcome data were considered and the inverse variance method was used for pooling. Sample sizes of individual trials were generated according to findings in a survey of eight German medical journals resulting in a large proportion of trials with sample size less than 100. A total of 10000 meta-analyses for each combination of the following factors was considered: odds-ratio/relative risk OR/RR in {0.5, 0.67, 1.0, 1.5, 2.0}; average event probability p in {0.1, 0.3}; number of trials in the meta-analysis k in {10,20,50}; between-trial variance tau^2 in {0%, 25%, 50%} given as the percentage of the within-trial variance of a trial with sample size 100. No bias in meta-analysis was induced, therefore, the proportion of significant findings should match the prespecified significance level of 10%.
Results: Simulation results indicate an inflation of type-I-error rates for both tests. For the odds-ratio, the estimated sizes are ranging from 11.1% to 28.2% (Egger et al. test) and from 9.5% to 29.1% (Begg and Mazumdar test). Results get worse with increasing treatment effect and number of trials combined. Furthermore, a tendency of anti-conservatism is apparent for the Egger test with increasing degree of heterogeneity.
Conclusions: In sparse data, the application of statistical tests for the detection of bias in meta-analysis results too often in the rejection of the null-hypothesis. Valid statistical tests for meta-analysis with sparse data have to be developed. References: 1. Begg, CB, Mazumdar, M (1994): Operating Characteristics of a Rank Correlation Test for Publication Bias. Biometrics, 50: 1088-1101. 2. Egger M, Smith GD, Schneider M, Minder C (1997): Bias in meta-analysis detected by a simple, graphical test. BMJ, 315: 629-634.
Methods: Binary outcome data were considered and the inverse variance method was used for pooling. Sample sizes of individual trials were generated according to findings in a survey of eight German medical journals resulting in a large proportion of trials with sample size less than 100. A total of 10000 meta-analyses for each combination of the following factors was considered: odds-ratio/relative risk OR/RR in {0.5, 0.67, 1.0, 1.5, 2.0}; average event probability p in {0.1, 0.3}; number of trials in the meta-analysis k in {10,20,50}; between-trial variance tau^2 in {0%, 25%, 50%} given as the percentage of the within-trial variance of a trial with sample size 100. No bias in meta-analysis was induced, therefore, the proportion of significant findings should match the prespecified significance level of 10%.
Results: Simulation results indicate an inflation of type-I-error rates for both tests. For the odds-ratio, the estimated sizes are ranging from 11.1% to 28.2% (Egger et al. test) and from 9.5% to 29.1% (Begg and Mazumdar test). Results get worse with increasing treatment effect and number of trials combined. Furthermore, a tendency of anti-conservatism is apparent for the Egger test with increasing degree of heterogeneity.
Conclusions: In sparse data, the application of statistical tests for the detection of bias in meta-analysis results too often in the rejection of the null-hypothesis. Valid statistical tests for meta-analysis with sparse data have to be developed. References: 1. Begg, CB, Mazumdar, M (1994): Operating Characteristics of a Rank Correlation Test for Publication Bias. Biometrics, 50: 1088-1101. 2. Egger M, Smith GD, Schneider M, Minder C (1997): Bias in meta-analysis detected by a simple, graphical test. BMJ, 315: 629-634.