Article type
Year
Abstract
Background: The recently developed Cochrane Update Classification System encourages reviewers to update systematic reviews (SRs) based on need instead of a time‐based frequency. The prioritisation involves that it is redundant to update SRs with conclusions so firm that new evidence is unlikely to change them. However, there is currently no methodological framework in place to support decision-making about the conclusiveness of evidence and, hence, for updating SRs with conclusive evidence.
Objectives: This study aimed to identify criteria which might have been used to mark SRs as being conclusive from a methodological point of view.
Methods: Based on the Update Classification System, we searched for SRs with conclusive evidence from the Gynaecology and Fertility, Airways, Eyes and Vision and Infectious Diseases Cochrane Review Groups in Archie. After screening of the rationale and explanation of the update status, SRs with conclusive evidence were selected based on assessment of the conclusions and recommendations for further research. Next, they were compared for different criteria (e.g. sample size, effect size, certainty of evidence,…) with a number of control SRs which are currently being updated.
Results: Out of 772 records, 8 SRs with conclusive evidence were included and 16 control SRs were selected. Although one would expect the evidence in SRs with conclusive evidence to be considered as high certainty according to the GRADE approach, we also identified outcomes of moderate or even very low certainty evidence. The main finding is that conclusions extracted from imprecise or indirect results were not considered as conclusive. However, inconsistency seems to be a less clear criterium since more than 50% of the conclusive outcomes suffer from substantial heterogeneity (I2 > 50%). We found that this statistical heterogeneity originated from differences in magnitude of effect and conclusive SRs never showed inconsistency from different directions of effect. Finally, publication bias and risk of bias seemed not to be distinguishing factors for marking SRs as conclusive. Risk of bias was tolerated when exclusion of studies at high or unclear risk of selection or detection bias did not change the result or when performance bias was inherent to the study design.
Conclusions: Imprecision, indirectness and inconsistency seem to be key for deciding whether a SR can be declared as conclusive about its effects on health. Decision makers might compromise on statistical heterogeneity, risk of bias and publication bias when these limitations do not reduce the confidence in the effect estimates. However, the update status should always be critically re-evaluated in the light of recent advancements in the field.
Patient or healthcare consumer involvement: Reviews declared as conclusive reveal that clinical research has identified interventions delivering meaningful patient-oriented outcomes. The caregivers, patients and funders can act with confidence.
Objectives: This study aimed to identify criteria which might have been used to mark SRs as being conclusive from a methodological point of view.
Methods: Based on the Update Classification System, we searched for SRs with conclusive evidence from the Gynaecology and Fertility, Airways, Eyes and Vision and Infectious Diseases Cochrane Review Groups in Archie. After screening of the rationale and explanation of the update status, SRs with conclusive evidence were selected based on assessment of the conclusions and recommendations for further research. Next, they were compared for different criteria (e.g. sample size, effect size, certainty of evidence,…) with a number of control SRs which are currently being updated.
Results: Out of 772 records, 8 SRs with conclusive evidence were included and 16 control SRs were selected. Although one would expect the evidence in SRs with conclusive evidence to be considered as high certainty according to the GRADE approach, we also identified outcomes of moderate or even very low certainty evidence. The main finding is that conclusions extracted from imprecise or indirect results were not considered as conclusive. However, inconsistency seems to be a less clear criterium since more than 50% of the conclusive outcomes suffer from substantial heterogeneity (I2 > 50%). We found that this statistical heterogeneity originated from differences in magnitude of effect and conclusive SRs never showed inconsistency from different directions of effect. Finally, publication bias and risk of bias seemed not to be distinguishing factors for marking SRs as conclusive. Risk of bias was tolerated when exclusion of studies at high or unclear risk of selection or detection bias did not change the result or when performance bias was inherent to the study design.
Conclusions: Imprecision, indirectness and inconsistency seem to be key for deciding whether a SR can be declared as conclusive about its effects on health. Decision makers might compromise on statistical heterogeneity, risk of bias and publication bias when these limitations do not reduce the confidence in the effect estimates. However, the update status should always be critically re-evaluated in the light of recent advancements in the field.
Patient or healthcare consumer involvement: Reviews declared as conclusive reveal that clinical research has identified interventions delivering meaningful patient-oriented outcomes. The caregivers, patients and funders can act with confidence.