Article type
Year
Abstract
Background: There are claims in the literature that the risk difference is a more heterogeneous measure than the odds ratio or risk ratio. These claims are based on surveys of meta-analyses showing that tests reject the null hypothesis of homogeneity more often for the risk difference than for ratio measures.
Objectives: To illustrate that differences in results of homogeneity tests across different scales may be related to statistical power rather than differences in homogeneity itself.
Methods: We use hypothetical examples where lack of homogeneity is arguably the same across different scales, but the power (and therefore the expected P value) for the different scales are remarkably different.
Results: In the first example, we simulated 75 participants per treatment group where the probability of outcome is 0.206 in Population A regardless of treatment. In Population B, treatment increased the probability of remission from 0.270 to 0.460. Simulating a meta-analysis based on these effects would yield a power of 47% for risk difference, but only 35% for the odds ratio. These numbers exactly match the results used by others to suggest the risk difference is more heterogeneous than the odds ratio. We also simulated 150 participants per group, where baseline remission probability was 0.2 in Population A and B. In this example, treatment increases probability of remission to 0.304 in Population A, and to 0.350 in Population B. but has an effect in Population B, leading to different risk differences and odds ratios in the two populations. These numbers also recreate a power of 47% for risk differences and only 35% for the odds ratios. The effect can also be reversed, with the test of homogeneity having less power for the risk difference compared to the odds ratio.
Conclusions: Because current methods cannot be used to conclude that one scale is more heterogeneous than another within meta-analyses of the same data, claims that the risk difference is more heterogeneous should be considered, at best, tentative. The meta-meta-analytic results, therefore, do not support a policy of routinely shunning any of the three measures, including the risk difference.
Objectives: To illustrate that differences in results of homogeneity tests across different scales may be related to statistical power rather than differences in homogeneity itself.
Methods: We use hypothetical examples where lack of homogeneity is arguably the same across different scales, but the power (and therefore the expected P value) for the different scales are remarkably different.
Results: In the first example, we simulated 75 participants per treatment group where the probability of outcome is 0.206 in Population A regardless of treatment. In Population B, treatment increased the probability of remission from 0.270 to 0.460. Simulating a meta-analysis based on these effects would yield a power of 47% for risk difference, but only 35% for the odds ratio. These numbers exactly match the results used by others to suggest the risk difference is more heterogeneous than the odds ratio. We also simulated 150 participants per group, where baseline remission probability was 0.2 in Population A and B. In this example, treatment increases probability of remission to 0.304 in Population A, and to 0.350 in Population B. but has an effect in Population B, leading to different risk differences and odds ratios in the two populations. These numbers also recreate a power of 47% for risk differences and only 35% for the odds ratios. The effect can also be reversed, with the test of homogeneity having less power for the risk difference compared to the odds ratio.
Conclusions: Because current methods cannot be used to conclude that one scale is more heterogeneous than another within meta-analyses of the same data, claims that the risk difference is more heterogeneous should be considered, at best, tentative. The meta-meta-analytic results, therefore, do not support a policy of routinely shunning any of the three measures, including the risk difference.