Article type
Year
Abstract
Background: While the randomized controlled trial (RCT) is generally regarded as the design of choice in health care, within public policy there is considerable debate about the suitability of RCTs and non-randomized studies (NRS). As the first stage of a study to explore effect sizes from comparable randomized and non-randomized policy evaluations, we conducted a systematic review of meta-analyses. Part Two of the study, which uses meta-regression, is reported separately (Thomas et al., submitted abstract).
Objectives: To search for, assess and synthesise meta-analyses of policy interventions that have analysed RCTs and NRS separately. To describe the methods used by reviewers to identify factors other than the use of randomization that may have influenced the results of RCTs and NRS.
Methods: Electronic databases were searched and meta-analyses screened for inclusion. Included meta-analyses underwent data extraction and critical appraisal using a web-based reviewing tool. Methods and results were tabulated and reported narratively.
Results: Sixteen meta-analyses were included. The scope of the studies was broad, with diversity in interventions and study populations. The majority of interventions aimed to promote health in institutional (e.g. hospitals or schools) or community settings. Few explicitly aimed to assess differences between study designs. RCTs and NRS were judged to have similar results in five meta-analyses, and different results in eight (of which five reported larger effects for NRS). In three meta-analyses similarities and differences in results varied according to outcomes. Many of the meta-analyses did not investigate the potential role of effect modifiers that may confound differences between designs.
Conclusions: This review of public policy interventions concurs with similar investigations in health care and in the social sciences, namely that RCTs and NRS are sometimes similar, and sometimes different in terms of results. In this sample, heterogeneity between designs makes it difficult to isolate the effect of randomization.
Objectives: To search for, assess and synthesise meta-analyses of policy interventions that have analysed RCTs and NRS separately. To describe the methods used by reviewers to identify factors other than the use of randomization that may have influenced the results of RCTs and NRS.
Methods: Electronic databases were searched and meta-analyses screened for inclusion. Included meta-analyses underwent data extraction and critical appraisal using a web-based reviewing tool. Methods and results were tabulated and reported narratively.
Results: Sixteen meta-analyses were included. The scope of the studies was broad, with diversity in interventions and study populations. The majority of interventions aimed to promote health in institutional (e.g. hospitals or schools) or community settings. Few explicitly aimed to assess differences between study designs. RCTs and NRS were judged to have similar results in five meta-analyses, and different results in eight (of which five reported larger effects for NRS). In three meta-analyses similarities and differences in results varied according to outcomes. Many of the meta-analyses did not investigate the potential role of effect modifiers that may confound differences between designs.
Conclusions: This review of public policy interventions concurs with similar investigations in health care and in the social sciences, namely that RCTs and NRS are sometimes similar, and sometimes different in terms of results. In this sample, heterogeneity between designs makes it difficult to isolate the effect of randomization.