Article type
Year
Abstract
Background: The GRADE Working Group provides an approach to assessing the quality of a body of evidence that is increasingly being used, involving explicit consideration of all sources of uncertainty. One of these sources is imprecision, or random error. Many currently published meta-analyses are underpowered and likely to be updated in the future. When data are sparse and there are repeated updates, the risk of random error is increased. Trial sequential analysis (TSA) is one of several methodologies that estimates this increased risk in meta-analyses. With nominally statistically significant meta-analyses of anaesthesia interventions, we used TSA to estimate power and imprecision in the context of sparse data and repeated updates.
Methods: We searched for systematic reviews with meta-analyses that investigated anaesthesia interventions.We randomly selected 50 meta-analyses that reported a statistically significant dichotomous outcome in their abstract. We applied TSA to these meta-analyses, using two main TSA approaches: relative risk reduction (RRR) 20% and RRR corresponding to the border of the conventional 95% confidence interval (CI) closest to the null. We calculated the power achieved by each included meta-analysis, using each TSA approach, and we calculated the proportion that maintained statistical significance when allowing for sparse data and repeated updates.
Results: In our sample, the median number of trials included was eight (interquartile range (IQR) 5 to 14), the median number of participants was 964 (IQR 523 to 1736) and the median number of events was 202 (IQR 96 to 443). Using both of our main TSA approaches, only 12% (95% CI 5% to 25%) of the meta-analyses had power ≥ 80%, and only 32% (95% CI 20% to 47%) of the meta-analyses preserved the risk of type I below 5%.
Conclusion: The majority of nominally statistically significant meta-analyses of anesthesia interventions are underpowered, and a large proportion do not maintain their risk of type 1 error below 5% if TSA monitoring boundaries are applied. A consideration of the effect of sparse data and repeated updates is needed when assessing imprecision in anaesthesia meta-analyses.
Methods: We searched for systematic reviews with meta-analyses that investigated anaesthesia interventions.We randomly selected 50 meta-analyses that reported a statistically significant dichotomous outcome in their abstract. We applied TSA to these meta-analyses, using two main TSA approaches: relative risk reduction (RRR) 20% and RRR corresponding to the border of the conventional 95% confidence interval (CI) closest to the null. We calculated the power achieved by each included meta-analysis, using each TSA approach, and we calculated the proportion that maintained statistical significance when allowing for sparse data and repeated updates.
Results: In our sample, the median number of trials included was eight (interquartile range (IQR) 5 to 14), the median number of participants was 964 (IQR 523 to 1736) and the median number of events was 202 (IQR 96 to 443). Using both of our main TSA approaches, only 12% (95% CI 5% to 25%) of the meta-analyses had power ≥ 80%, and only 32% (95% CI 20% to 47%) of the meta-analyses preserved the risk of type I below 5%.
Conclusion: The majority of nominally statistically significant meta-analyses of anesthesia interventions are underpowered, and a large proportion do not maintain their risk of type 1 error below 5% if TSA monitoring boundaries are applied. A consideration of the effect of sparse data and repeated updates is needed when assessing imprecision in anaesthesia meta-analyses.