Article type
Year
Abstract
Introduction/Objective: To evaluate the performance of standard methods of meta-analysis of binary data when event rates are very low. An issue that arises when studying uncommon outcomes in trials is that a substantial proportion of studies may report no events in either the treated group, or the control group, or both. Zero cells in contingency tables cause problems, both in terms of the validity of the methods when numbers are small, and the practicalities of coping with possible divisions by zero. Most traditional methods based on ratio measures, such as odds ratios (OR), ignore studies in which no events occur in either group.
Methods: We conducted a series of statistical simulations in which data for two study groups, with known, low event probabilities, were generated. Relative risks of 1.0, 0.75 and 0.5 were considered corresponding to varying strengths of treatment benefit. Meta-analyses of 5 and 20 studies were simulated, the sample sizes being based on the results of real Cochrane reviews, with baseline (control group) event rates of 5%, 1%, 0.5% and 0.1%. The performance of a selection of the statistical methods available in RevMan (Mantel-Haenszel (MH) OR with RBG variance, Peto OR, D&L random effects OR, and the MH and D&L risk difference methods) together with other methods not currently available in RevMan (Poisson regression models, inverse variance and exact methods) was evaluated. At each combination of relative risk and baseline event rate 10,000 meta-analyses were simulated. Each method was evaluated with respect to bias and statistical power.
Results/Discussion: For datasets generated assuming a fixed underlying effect, contrary to current expectations, the Peto method provided the least biased and most powerful method of pooling study results among the methods available in RevMan. Other methods tended to underestimate treatment effects by between 25% and 35% when event rates were 0.1% and the treatment effect was a RR=0.5. The bias increased with decreasing baseline rates, increasing treatment effects and increasing imbalance in trial group sizes. Conclusions: Several of the statistical methods available in RevMan perform very poorly when event rates are low, and tend to underestimate treatment effects. The Peto method may be the best method in these circumstances unless trial group sizes are severely imbalanced, and will provide a good approximation to the relative risk.
Methods: We conducted a series of statistical simulations in which data for two study groups, with known, low event probabilities, were generated. Relative risks of 1.0, 0.75 and 0.5 were considered corresponding to varying strengths of treatment benefit. Meta-analyses of 5 and 20 studies were simulated, the sample sizes being based on the results of real Cochrane reviews, with baseline (control group) event rates of 5%, 1%, 0.5% and 0.1%. The performance of a selection of the statistical methods available in RevMan (Mantel-Haenszel (MH) OR with RBG variance, Peto OR, D&L random effects OR, and the MH and D&L risk difference methods) together with other methods not currently available in RevMan (Poisson regression models, inverse variance and exact methods) was evaluated. At each combination of relative risk and baseline event rate 10,000 meta-analyses were simulated. Each method was evaluated with respect to bias and statistical power.
Results/Discussion: For datasets generated assuming a fixed underlying effect, contrary to current expectations, the Peto method provided the least biased and most powerful method of pooling study results among the methods available in RevMan. Other methods tended to underestimate treatment effects by between 25% and 35% when event rates were 0.1% and the treatment effect was a RR=0.5. The bias increased with decreasing baseline rates, increasing treatment effects and increasing imbalance in trial group sizes. Conclusions: Several of the statistical methods available in RevMan perform very poorly when event rates are low, and tend to underestimate treatment effects. The Peto method may be the best method in these circumstances unless trial group sizes are severely imbalanced, and will provide a good approximation to the relative risk.