Article type
Year
Abstract
Background: systematic reviews offer the most reliable and valid support for health policy decision making, patient information, and guideline development. Conducting systematic reviews, however, is time and labour intensive. Because of the time-sensitive needs of decision makers, accelerating the production of evidence syntheses has become a relevant issue. One possibility to save time and resources is to limit reviews to English-language references. This approach, however, risks introducing language bias. To date the impact of this approach has been assessed on single outcomes of systematic reviews but not on overall conclusions across multiple bodies of evidence.
Objective: we aimed to assess the effects of limiting systematic reviews to English-language records on the overall conclusions.
Methods: our analyses used a dataset of a previous methods study that included 59 randomly selected Cochrane Reviews on clinical interventions without language restrictions. We checked the publication language of all 2026 included references. A study was classified as missed if 1) the only reference to a study was non-English, or 2) the non-English reference was the main publication in case of multiple references to the same study. We re-calculated meta-analyses for outcomes that were presented in the main 'Summary of findings' tables of the Cochrane Reviews. If the effect estimate of an outcome changed direction or statistical significance, we asked authors of the respective Cochrane Review to assess whether the new evidence base would change their conclusions in the Cochrane Review. We used a non-inferiority design with the primary outcome: proportion of conclusions that would change when excluding non-English studies. We regarded the approach as non-inferior if the upper limit of the 95% confidence interval of the proportion of changed conclusions was the same or less than the non-inferiority margin of 10%, which we determined based on the results of a survey for clinical and public health scenarios.
Results: across all 59 Cochrane Reviews, 80 (4%) out of 2026 references were non-English. Twenty-nine (49%) Cochrane Reviews included non-English references. Eliminating non-English references from the evidence base, excluded 31 studies (1 to 8 studies per review) in 16 Cochrane Reviews (27%). The proportion of changed conclusions was 0.0% (95% CI 0.00 to 0.10). Only in two cases would authors have drawn a conclusion with less certainty, albeit still in the same direction as the original conclusion.
Conclusion: excluding non-English records from systematic reviews on clinical interventions had a minimal effect on overall conclusions and could be a viable methodological shortcut especially for rapid reviews.
Objective: we aimed to assess the effects of limiting systematic reviews to English-language records on the overall conclusions.
Methods: our analyses used a dataset of a previous methods study that included 59 randomly selected Cochrane Reviews on clinical interventions without language restrictions. We checked the publication language of all 2026 included references. A study was classified as missed if 1) the only reference to a study was non-English, or 2) the non-English reference was the main publication in case of multiple references to the same study. We re-calculated meta-analyses for outcomes that were presented in the main 'Summary of findings' tables of the Cochrane Reviews. If the effect estimate of an outcome changed direction or statistical significance, we asked authors of the respective Cochrane Review to assess whether the new evidence base would change their conclusions in the Cochrane Review. We used a non-inferiority design with the primary outcome: proportion of conclusions that would change when excluding non-English studies. We regarded the approach as non-inferior if the upper limit of the 95% confidence interval of the proportion of changed conclusions was the same or less than the non-inferiority margin of 10%, which we determined based on the results of a survey for clinical and public health scenarios.
Results: across all 59 Cochrane Reviews, 80 (4%) out of 2026 references were non-English. Twenty-nine (49%) Cochrane Reviews included non-English references. Eliminating non-English references from the evidence base, excluded 31 studies (1 to 8 studies per review) in 16 Cochrane Reviews (27%). The proportion of changed conclusions was 0.0% (95% CI 0.00 to 0.10). Only in two cases would authors have drawn a conclusion with less certainty, albeit still in the same direction as the original conclusion.
Conclusion: excluding non-English records from systematic reviews on clinical interventions had a minimal effect on overall conclusions and could be a viable methodological shortcut especially for rapid reviews.
PDF