Article type
Year
Abstract
Background: Studies, mainly clinical, have identified design features of rigorous RCTs that reduce systematic bias in estimating effect sizes but this evidence base is lacking for public policy evaluation. A review of methodological literature and a systematic review of systematic reviews of the effects of policy interventions found that RCTs and non-RCTs lead to different effect sizes (Shepherd et al., submitted abstract). A number of hypotheses but little empirical evidence explained these differences.
Objectives: To examine whether RCTs of policy interventions produce significantly different results when compared with other study designs or whether heterogeneity, if found, can be explained by other factors.
Methods: We investigated theoretical differences in effect size associated with randomization by:
-comparing controlled trials that are identical in all respects other than the use of randomization by 'breaking' the randomization in a trial to create non-randomized trials and smaller randomized trials.
We investigated differences encountered with field trials by:
- comparing similar controlled trials drawn from systematic reviews that include both randomized and non-randomized studies;
- investigating associations between randomization and effect size using a pool of more diverse studies within broadly similar areas.
All studies were coded for characteristics of the population, policy intervention and evaluation. Associations between these characteristics, randomization and effect size were tested by chi-square calculations. To allow for unexplained heterogeneity between studies as well as the known uncertainty in estimated effect sizes (measured by their standard errors), random-effects meta-regression was used.
Results: Trials which are identical in all respects except randomization lead to the same effect size, but sometimes with greater variance. In the field, effect sizes can differ and extensive empirical investigations fail to explain them.
Conclusions: Despite the challenges of randomization for large policy evaluations, no other method can be recommended for
minimising selection bias when assessing effectiveness.
Objectives: To examine whether RCTs of policy interventions produce significantly different results when compared with other study designs or whether heterogeneity, if found, can be explained by other factors.
Methods: We investigated theoretical differences in effect size associated with randomization by:
-comparing controlled trials that are identical in all respects other than the use of randomization by 'breaking' the randomization in a trial to create non-randomized trials and smaller randomized trials.
We investigated differences encountered with field trials by:
- comparing similar controlled trials drawn from systematic reviews that include both randomized and non-randomized studies;
- investigating associations between randomization and effect size using a pool of more diverse studies within broadly similar areas.
All studies were coded for characteristics of the population, policy intervention and evaluation. Associations between these characteristics, randomization and effect size were tested by chi-square calculations. To allow for unexplained heterogeneity between studies as well as the known uncertainty in estimated effect sizes (measured by their standard errors), random-effects meta-regression was used.
Results: Trials which are identical in all respects except randomization lead to the same effect size, but sometimes with greater variance. In the field, effect sizes can differ and extensive empirical investigations fail to explain them.
Conclusions: Despite the challenges of randomization for large policy evaluations, no other method can be recommended for
minimising selection bias when assessing effectiveness.