Article type
Year
Abstract
Background: Low quality trials may find larger treatment effects due to bias. Several studies have assessed the association between quality components and treatment effect. Results are conflicting.
Objectives: To review the results of empirical studies on the association between methodological quality components and treatment effects in randomised trials.
Methods: We included six empirical studies assessing the association between at least one of five methodological quality components and treatment effects. The quality components were: generation of the allocation sequence, allocation concealment, double blinding, intention-to-treat analyses, and power calculation. Quality components were categorised as adequate or inadequate (including unclear). For each quality component, we meta-analysed the ratio of odds ratios (RORs) comparing treatment effects in trials with inadequate to trials with adequate quality components. RORs less than 1 indicate larger treatment effects in trials with inadequate description of a given quality component. Meta-analyses were performed using random effects models. Statistical heterogeneity was explored by chi-squared tests with significance set at P < 0.1. The inconsistency across trials was assessed by I2.
Results: The treatment effect estimates were significantly larger in trials with inadequate compared to adequate generation of the allocation sequence (ROR 0.88, 95%CI 0.79-0.99, five studies) and allocation concealment (ROR 0.79, 95% CI 0.66-0.95, six studies, figure 1), but there was considerable heterogeneity among study results (generation: P = 0.05, I2 = 58.3%; concealment: P < 0.00001, I2 = 88.5%). Inadequate double blinding was not significantly associated with larger treatment effects (ROR 0.82, 95% CI 0.71-1.05, six studies), nor was the lack of intention-to-treat analyses (ROR 1.06, 95% CI 0.92-1.22, two studies) or reporting of power calculations (ROR 0.95, 95% CI 0.88-1.03, two studies).
Conclusions: Inadequate randomisation procedures seem to be associated with larger estimates of treatment effect, but there is substantial heterogeneity among the results of empirical studies. The impact of bias seems to vary considerably across interventions and disease areas.
Objectives: To review the results of empirical studies on the association between methodological quality components and treatment effects in randomised trials.
Methods: We included six empirical studies assessing the association between at least one of five methodological quality components and treatment effects. The quality components were: generation of the allocation sequence, allocation concealment, double blinding, intention-to-treat analyses, and power calculation. Quality components were categorised as adequate or inadequate (including unclear). For each quality component, we meta-analysed the ratio of odds ratios (RORs) comparing treatment effects in trials with inadequate to trials with adequate quality components. RORs less than 1 indicate larger treatment effects in trials with inadequate description of a given quality component. Meta-analyses were performed using random effects models. Statistical heterogeneity was explored by chi-squared tests with significance set at P < 0.1. The inconsistency across trials was assessed by I2.
Results: The treatment effect estimates were significantly larger in trials with inadequate compared to adequate generation of the allocation sequence (ROR 0.88, 95%CI 0.79-0.99, five studies) and allocation concealment (ROR 0.79, 95% CI 0.66-0.95, six studies, figure 1), but there was considerable heterogeneity among study results (generation: P = 0.05, I2 = 58.3%; concealment: P < 0.00001, I2 = 88.5%). Inadequate double blinding was not significantly associated with larger treatment effects (ROR 0.82, 95% CI 0.71-1.05, six studies), nor was the lack of intention-to-treat analyses (ROR 1.06, 95% CI 0.92-1.22, two studies) or reporting of power calculations (ROR 0.95, 95% CI 0.88-1.03, two studies).
Conclusions: Inadequate randomisation procedures seem to be associated with larger estimates of treatment effect, but there is substantial heterogeneity among the results of empirical studies. The impact of bias seems to vary considerably across interventions and disease areas.