Background:

The extent of disagreement between large and smaller RCTs on mental health issues is unknown.

Objective:

We aimed to compare the results of large vs. smaller trials on mental health related interventions.

Methods:

We screened 161 Cochrane and 254 DARE systematic reviews on mental health related interventions. Eligible were meta-analyses with at least 1 large randomized trial (RCT) with sample size >800, and at least 1 smaller trial. Effect sizes were calculated separately for large and smaller trials. RCTs in each meta-analysis were further scrutinized with respect to the presence or absence of three quality items: double blinding, adequate mode of randomization and adequate allocation concealment. Heterogeneity was assessed between all studies in each meta-analysis, within each group of large and smaller studies per meta-analysis, and across large and smaller studies per meta-analysis.

Results:

Sixteen qualifying meta-analyses were identified with 133 RCTs included (small n=101, large n=32). Significant between-study heterogeneity was seen in 5 meta-analyses. Disagreements in the effect sizes between large and smaller trials, beyond what would have been expected by chance alone, was noted in 4 meta-analyses (25%). In 3 of these disagreements, the smaller trials showed greater effect sizes than the large trials. The inverse occurred in 1 case. By fixed effects models, disagreements beyond chance occurred in 31% (5/16) of meta-analyses. In 4 meta-analyses, the effect size differed over 2-fold between large and smaller trials. Of the 133 RCTs, 56 were double blind, but only 18 reported an adequate mode of randomization and only 7 reported adequate allocation concealment. Direct comparison of RCTs in each meta-analysis with or without these 3 quality related items did not reveal statistically significant discrepancies in the summary effect size with 2 exceptions. Trials with adequate mode of randomization or allocation concealment did not have consistently more conservative results than other
trials.

Conclusions:

Large trials are uncommon in mental health. Their results are usually comparable with the results of smaller studies, but major disagreements do occur. Differences based on quality items were not consistent. Both large and smaller trials should be scrutinized as they provide a continuum of evidence.