Article type
Year
Abstract
Background:
Cochrane Reviews require updating as new studies become available, requiring multiple, repeated meta-analyses to analyse these studies. This increases the risk of attaining spurious 'statistical significance' in each analysis. As updating of reviews becomes both more common and more frequent there is an increasing risk that meta-analysis results may be misinterpreted, particularly if readers are unaware of the updating process.
Objectives:
To discuss the findings of a Cochrane-funded project, which has formally assessed a number of statistical methods for appropriately updating meta-analyses.
Methods:
Methods assessed include standard random-effects analysis, trial sequential analysis and sequential meta-analysis. These have been applied to a range of recently updated Cochrane Reviews and tested in simulation studies.
Results:
The use of standard meta-analysis methods when updating Cochrane Reviews does increase the risk of spurious findings, which would be overturned at later updates. This risk is substantial when a meta-analysis includes few trials, or the trials are very heterogeneous in their findings. Trial sequential analysis and sequential meta-analysis, and potentially Bayesian analyses, are the only statistical methods found to reduce this risk. This benefit comes at the cost of requiring more trials or review updates to reach a firm conclusion.
Conclusions:
All Cochrane Reviews should consider the risk of drawing incorrect conclusions due to repeated updating and analysis over time, particularly where reviews are updated frequently. One simple option is to consider whether the total sample size of the review is adequate. Trial sequential analysis and sequential meta-analysis offer formal statistical techniques for reliable updating of reviews, but come with greater statistical complexity that may be unnecessary in reviews with infrequent updates. These methods may be of particular importance where reviews guide clinical decision-making, where having robust evidence that an intervention is useful is essential.
Patient or healthcare consumer involvement:
This project was funded by the Cochrane Methods Innovation Fund and had no direct patient involvement. However, patients rely on reviews being up-to-date, and that medical decisions based on reviews are justified given the evidence and will not change in the future.
Cochrane Reviews require updating as new studies become available, requiring multiple, repeated meta-analyses to analyse these studies. This increases the risk of attaining spurious 'statistical significance' in each analysis. As updating of reviews becomes both more common and more frequent there is an increasing risk that meta-analysis results may be misinterpreted, particularly if readers are unaware of the updating process.
Objectives:
To discuss the findings of a Cochrane-funded project, which has formally assessed a number of statistical methods for appropriately updating meta-analyses.
Methods:
Methods assessed include standard random-effects analysis, trial sequential analysis and sequential meta-analysis. These have been applied to a range of recently updated Cochrane Reviews and tested in simulation studies.
Results:
The use of standard meta-analysis methods when updating Cochrane Reviews does increase the risk of spurious findings, which would be overturned at later updates. This risk is substantial when a meta-analysis includes few trials, or the trials are very heterogeneous in their findings. Trial sequential analysis and sequential meta-analysis, and potentially Bayesian analyses, are the only statistical methods found to reduce this risk. This benefit comes at the cost of requiring more trials or review updates to reach a firm conclusion.
Conclusions:
All Cochrane Reviews should consider the risk of drawing incorrect conclusions due to repeated updating and analysis over time, particularly where reviews are updated frequently. One simple option is to consider whether the total sample size of the review is adequate. Trial sequential analysis and sequential meta-analysis offer formal statistical techniques for reliable updating of reviews, but come with greater statistical complexity that may be unnecessary in reviews with infrequent updates. These methods may be of particular importance where reviews guide clinical decision-making, where having robust evidence that an intervention is useful is essential.
Patient or healthcare consumer involvement:
This project was funded by the Cochrane Methods Innovation Fund and had no direct patient involvement. However, patients rely on reviews being up-to-date, and that medical decisions based on reviews are justified given the evidence and will not change in the future.