Background: In meta-analysis of continuous outcomes, the most commonly used effect measure is the difference in means, either directly [mean difference (MD)] or divided by the pooled standard deviation [standardized mean difference (SMD)]. We recently used a new method to meta-analyze continuous outcomes by calculating a ratio of mean values (RoM) instead of a difference for each study. We estimated the variance of this ratio using the delta method and used inverse variance weighting to pool ratios. Both SMD and RoM but not MD allow pooling of studies with outcomes expressed in different units and comparisons of effect sizes across different interventions. Interpretation of SMD requires knowledge of the pooled standard deviation (SD), a quantity generally unknown to clinicians. In contrast, this information is not required when using RoM to estimate the expected treatment effect for a particular patient.
Objectives: To test the hypothesis that MD, SMD and RoM exhibit comparable performance characteristics using simulation.
Methods: Parameter values used to simulate data sets (SAS version 8.2) were chosen to be representative of those commonly encountered in meta-analyses of continuous outcomes: effect sizes (0.2, 0.5, or 0.8 pooled SD units to represent small, medium, and large effect sizes), SD (10, 40, or 70% of the mean value to reflect a narrow, medium and broad distribution), number of trials (5, 10, or 30), number of control and experimental participants per trial group (10 or 100), and heterogeneity (I2=0 or 50-95%). The simulations used equal standard deviations for the control and experimental groups and were repeated 10,000 times for each scenario.
Results: Whereas the MD method was relatively free of bias (<1.5%) for all scenarios, the SMD method exhibited negative bias (around 5%) in the scenarios with few patients (n=10), as described previously. The RoM method was free of bias (<1.5%) except for some scenarios with a broad distribution and medium to large effect sizes. There was a negative bias ranging from 1.6 to 3.8% for such scenarios with 10 patients per trial group and 10 or 30 trials, and a positive bias of around 2% for such scenarios with 100 patients per trial group, 5 trials and significant heterogeneity (I2>90%). The proportion of the scenarios for which the 95% confidence interval contained the true effect size (i.e. coverage) was identical for all scenarios with minimal bias. The coverage was as expected (i.e. close to 95%) for the scenarios with no heterogeneity, but decreased when heterogeneity was introduced. RoM scenarios with 30 trials and negative bias exceeding 1.5% (discussed above) demonstrated lower coverage than MD (89-92% vs. 94%). Coverage of RoM was also slightly lower than MD (90-92% vs. 92-94%) with broad distribution, medium to large effect sizes, 100 patients per trial group, 10 or 30 trials, and significant heterogeneity. The proportion of the scenarios that yield significant treatment effects (i.e. statistical power) was identical for all scenarios with minimal bias. Scenarios with negative bias demonstrated decreased statistical power. Compared to the MD method, simulated heterogeneity estimates for the SMD and RoM methods were lower in the scenarios in which they exhibited bias. This is because the bias decreased the weighting of extreme values. In the scenarios exhibiting minimal bias, heterogeneity was similar among methods.
Conclusions: Simulation suggests that the performance characteristics (bias, coverage, statistical power) of the RoM method compares favorably to the traditionally used MD and SMD methods. Similar to binary outcome analysis, this straightforward method provides researchers the option of using a ratio method which may be more interpretable by clinicians.