Article type
Year
Abstract
Background: Most meta-analyses include data from one or more small studies, which would not themselves have power to detect an intervention effect. The relative influence of adequately powered and underpowered studies in published meta-analyses has not previously been explored.
Objectives: To examine the amount of power available in studies included in meta-analyses reported by Cochrane reviews, and to investigate the impact of underpowered studies on meta-analysis results.
Methods: Our analyses included 14,886 meta-analyses of binary outcomes extracted from 1,991 Cochrane reviews. For each study in each meta-analysis, we calculated the power available to detect a 30% relative risk reduction, based on mean study arm size and prevalence of the outcome. Associations between meta-analysis characteristics and power were examined. In a subset of the data set, meta-analyses were repeated with underpowered studies excluded.
Results: In 81% of meta-analyses in this data set, all studies included were underpowered (power<50%) to detect a relative risk reduction of 30%. Only 10% of meta-analyses included at least two adequately powered studies (power≥50%). The amount of power typically available varied across medical areas and outcome types. In meta-analyses including at least two adequately powered studies and at least one underpowered study, differences between results based on adequately powered studies alone and results based on all studies were small on average. The standard error of the intervention effect increased by a median of 12% (inter-quartile (IQ) range -1% to 35%) when underpowered studies were omitted. The between-study heterogeneity estimate decreased by a median of 27% (IQ range 100% decrease to 14% increase).
Conclusions: In the majority of meta-analyses reported by Cochrane reviews, underpowered studies make up the entirety of the evidence. However, for topics in which at least two adequately powered studies are available, underpowered studies often contribute little information, which raises questions about scientific efficiency.
Objectives: To examine the amount of power available in studies included in meta-analyses reported by Cochrane reviews, and to investigate the impact of underpowered studies on meta-analysis results.
Methods: Our analyses included 14,886 meta-analyses of binary outcomes extracted from 1,991 Cochrane reviews. For each study in each meta-analysis, we calculated the power available to detect a 30% relative risk reduction, based on mean study arm size and prevalence of the outcome. Associations between meta-analysis characteristics and power were examined. In a subset of the data set, meta-analyses were repeated with underpowered studies excluded.
Results: In 81% of meta-analyses in this data set, all studies included were underpowered (power<50%) to detect a relative risk reduction of 30%. Only 10% of meta-analyses included at least two adequately powered studies (power≥50%). The amount of power typically available varied across medical areas and outcome types. In meta-analyses including at least two adequately powered studies and at least one underpowered study, differences between results based on adequately powered studies alone and results based on all studies were small on average. The standard error of the intervention effect increased by a median of 12% (inter-quartile (IQ) range -1% to 35%) when underpowered studies were omitted. The between-study heterogeneity estimate decreased by a median of 27% (IQ range 100% decrease to 14% increase).
Conclusions: In the majority of meta-analyses reported by Cochrane reviews, underpowered studies make up the entirety of the evidence. However, for topics in which at least two adequately powered studies are available, underpowered studies often contribute little information, which raises questions about scientific efficiency.