Article type
Abstract
Background: Investigations of replicability (the extent to which results differ when a prior study is repeated) have been directed largely at individual studies. It is just as critical to explore these issues in systematic reviews of studies, given their influence on decision-making and future research.
Objectives: To evaluate the extent of variation in results when we independently replicate the searches for studies, collection of results data, and meta-analysis of results in a sample of original reviews
Methods: We randomly selected 10 systematic reviews of the effects of interventions indexed in various databases in November 2020, in which methods and results of the searches and meta-analyses were reported completely. Two information specialists reran the original search strategies (limited to the original time frame) and recorded the number of records retrieved. Two investigators independently collected from each study report the results data needed to replicate the first meta-analysis and calculated meta-analysis results (ie, a meta-analytic effect estimate, its 95% confidence interval and P value, and measures of heterogeneity). We calculated for each database search how often the difference between the original and replicated number of records yielded was greater than 10%. We compared the original and recalculated meta-analysis results and judged whether the observed difference was meaningful (ie, change in statistical significance or direction).
Results: Across the 10 systematic reviews, 46 bibliographic database searches were rerun (median (IQR) of 5 (3.5, 5.75) per review). The difference between the original and replicated number of records yielded was greater than 10% for 26 (57%) databases. Of the first 8 meta-analyses that have been replicated, results were not fully replicable in 5 cases, and the difference between the original and replicated meta-analysis results was considered meaningful in 4 cases.
Conclusions: Our preliminary results suggest that the findings of original systematic reviews are not always consistent when the methods are repeated. Greater clarity and detail about the search methods, study eligibility criteria, and methods for selecting study results may enhance replicability of systematic reviews.
Relevance and importance to the public: Our findings will inform strategies to improve replicability, and thereby the public’s trust, in systematic reviews.
Objectives: To evaluate the extent of variation in results when we independently replicate the searches for studies, collection of results data, and meta-analysis of results in a sample of original reviews
Methods: We randomly selected 10 systematic reviews of the effects of interventions indexed in various databases in November 2020, in which methods and results of the searches and meta-analyses were reported completely. Two information specialists reran the original search strategies (limited to the original time frame) and recorded the number of records retrieved. Two investigators independently collected from each study report the results data needed to replicate the first meta-analysis and calculated meta-analysis results (ie, a meta-analytic effect estimate, its 95% confidence interval and P value, and measures of heterogeneity). We calculated for each database search how often the difference between the original and replicated number of records yielded was greater than 10%. We compared the original and recalculated meta-analysis results and judged whether the observed difference was meaningful (ie, change in statistical significance or direction).
Results: Across the 10 systematic reviews, 46 bibliographic database searches were rerun (median (IQR) of 5 (3.5, 5.75) per review). The difference between the original and replicated number of records yielded was greater than 10% for 26 (57%) databases. Of the first 8 meta-analyses that have been replicated, results were not fully replicable in 5 cases, and the difference between the original and replicated meta-analysis results was considered meaningful in 4 cases.
Conclusions: Our preliminary results suggest that the findings of original systematic reviews are not always consistent when the methods are repeated. Greater clarity and detail about the search methods, study eligibility criteria, and methods for selecting study results may enhance replicability of systematic reviews.
Relevance and importance to the public: Our findings will inform strategies to improve replicability, and thereby the public’s trust, in systematic reviews.