Background: When pooling intervention effects calculated from continuous data, review authors will commonly encounter estimates of intervention effect calculated from a variety of methods, a small number of randomized controlled trials (RCTs) with some or all including few participants, inconsistency of intervention effects between studies, and heterogeneity of variances within RCTs. It is currently unknown how meta-analytical methods perform under these scenarios.

Objectives: To examine the performance of fixed-effect and random-effects meta-analysis (MA) models for pooling intervention estimates calculated from continuous data under commonly encountered scenarios.

Methods: We present results of a simulation study which assessed the performance measures: Type I error rate, bias, power, and coverage, for meta-analyses with few trials and participants, under a range of scenarios. We examined if the performance was dependent on varying degrees of between-study heterogeneity, the combination of analytical methods used to analyse the RCTs (selection mechanism), and the following within RCT factors: heterogeneity of variances, correlation between baseline and follow up, and size of the intervention effect. Simulation parameters were based on characteristics of published RCTs where possible.

Results: Inflated Type I error rates were observed when fixed-effect MA was used to pool intervention estimates. Some selection mechanisms resulted in biased estimates of intervention effect. The level of bias was invariant to the size of intervention effect, to the between-study heterogeneity, and to the meta-analytical model used. Power was higher for fixed-effect compared to random-effects models. However, coverage was greater for random-effects models.

Conclusions: The selection of analysis method used in RCTs can result in biased estimates of pooled intervention effect. Random-effects MA, in general, should be the preferred meta-analytical model for continuous data with few RCTs and participants.