Article type
Year
Abstract
Background:
Systematic reviews are resource-intensive so it is important to focus on reviewing interventions for which new evidence might warrant a change in practice.
Objectives:
To determine whether basic information about new relevant trials can be used to estimate the risk of a change in conclusion in published systematic reviews.
Methods:
We identified systematic reviews that had updates published between October 2016 and December 2017, including pairs with consistent search strategies, inclusion criteria and outcomes, and where most included studies were trials. We analysed reviews that added new trials and reported the numbers of participants. We extracted: the total number of trials and participants in the original review; the time between the two search dates; and the completeness - the number of participants in the original review as a proportion of the number of participants in the update. A change in conclusion was defined by a change in significance of a primary safety or efficacy outcome (evaluated independently by two investigators; disagreements resolved by discussion). We trained a Classification and Regression Tree to predict (five-fold cross validation) a change in conclusion using some or all of the factors, reporting average precision and recall.
Results:
We analysed 63 pairs of reviews, of which 20 reported a change in conclusion in the update. Using the number of trials/participants in the original review and time elapsed to the new search date, the decision tree produced an average precision of 40% and a recall of 70%. After adding completeness to the decision tree, this increased to an average precision of 60% and a recall of 90%. The decision tree showed that reviews were most at risk of a change in conclusion when completeness was low (≤ 13.5%), the original review had fewer trials (< 23) and more time had elapsed (> 53 months).
Conclusions:
An empirically defined decision tree using simple information extracted from a published systematic review and basic information about trials that may be relevant can estimate the risk of a change in conclusion. The results can be used to better target resources for updating systematic reviews and would benefit patients by identifying evidence reversals earlier.
Systematic reviews are resource-intensive so it is important to focus on reviewing interventions for which new evidence might warrant a change in practice.
Objectives:
To determine whether basic information about new relevant trials can be used to estimate the risk of a change in conclusion in published systematic reviews.
Methods:
We identified systematic reviews that had updates published between October 2016 and December 2017, including pairs with consistent search strategies, inclusion criteria and outcomes, and where most included studies were trials. We analysed reviews that added new trials and reported the numbers of participants. We extracted: the total number of trials and participants in the original review; the time between the two search dates; and the completeness - the number of participants in the original review as a proportion of the number of participants in the update. A change in conclusion was defined by a change in significance of a primary safety or efficacy outcome (evaluated independently by two investigators; disagreements resolved by discussion). We trained a Classification and Regression Tree to predict (five-fold cross validation) a change in conclusion using some or all of the factors, reporting average precision and recall.
Results:
We analysed 63 pairs of reviews, of which 20 reported a change in conclusion in the update. Using the number of trials/participants in the original review and time elapsed to the new search date, the decision tree produced an average precision of 40% and a recall of 70%. After adding completeness to the decision tree, this increased to an average precision of 60% and a recall of 90%. The decision tree showed that reviews were most at risk of a change in conclusion when completeness was low (≤ 13.5%), the original review had fewer trials (< 23) and more time had elapsed (> 53 months).
Conclusions:
An empirically defined decision tree using simple information extracted from a published systematic review and basic information about trials that may be relevant can estimate the risk of a change in conclusion. The results can be used to better target resources for updating systematic reviews and would benefit patients by identifying evidence reversals earlier.