It is increasingly common to see analyses of treatment efficacy from randomized trials as a function of the prevalence of the event rate in the control group. These analyses are of control prevalence strata within a single trial or across a set of trials within the context of a meta-analysis. We have identified two sources of bias which explain why observations that treatment has greater efficacy in high risk populations may be biased. In a Monte Carlo study, we demonstrate that these biases are always negative (treatment efficacy always appears to increase with increases in control group prevalence) and strong (actual decreasing efficacy with increased control prevalence could appear reversed). We illustrate the magnitude of bias using a published meta analysis of tocolysis with b-mimetics on preterm birth.