Are trial size and reported methodological quality associated with treatment effects? Observational study of 523 randomised trials

Als-Nielsen B, Chen W, Lotte Gluud L, Siersma V, Hilden J, Gluud C

Objectives: The objective of this study was to assess whether the size of a trial and five methodological quality components were associated with the estimated treatment effects in randomised trials.

Methods: Observational study of 523 randomised trials included in meta-analyses from Cochrane reviews selected from The Cochrane Library, Issue 2, 2001. From a random sample of 167 reviews, 41 contained 48 eligible meta-analyses (assessed a binary outcome; pooled at least five full-paper trials of which at least one reported adequate and one reported inadequate allocation concealment). Trials were classified into six categories according to the type of experimental and control intervention. The quality components were: generation of the allocation sequence, allocation concealment, double blinding, intention-to-treat analyses, and power calculation. Quality components were dichotomized into adequate and inadequate (including unclear). For each quality component, we calculated ratio of odds ratios (RORs) and 95% confidence intervals (CI) comparing treatment effects in inadequate with adequate quality trials. RORs less than 1 indicate larger treatment effect in trials with inadequate description of a given quality component. As inadequate quality components frequently co-occur, we performed both univariate and multivariate analyses using an intermediate approach (Siersma et al, Cochrane Colloquium, Ottawa 2004).

Results: The trial size, entered as precision in the model, was significantly associated with the estimated treatment effect in both univariate and multivariate analyses: the smaller the trial the larger the effect (Table 1). In the univariate analyses, only inadequate generation of the allocation sequence was significantly associated with a larger treatment effect. In the multivariate model, there was less evidence of this association. The other four quality components were not significantly associated with treatment effect (Table 1).

Conclusions: Small trials find larger treatment effects. This may be due to publication or other biases. Although there is a logical basis for suspecting bias in low quality trials, individual quality components were not consistently associated with treatment effect across trials.