Article type
Year
Abstract
Background: Despite the intention of a systematic review (SR) to identify and evaluate as much relevant evidence as possible with as little bias as possible, it is common for English-language limits to be applied at searching or screening stage.
Objectives: The objective of our research is to see how many SRs which do not use English-language limits, go on to have non-English included studies. We want to investigate connections between an SR which does not impose language restrictions and its risk of bias in Domain 2 (D2) of the ROBIS (Risk of Bias) tool (Identification and Selection of Studies). We also want to see if there are patterns in relation to topic and the inclusion of non-English studies which might indicate when English-language limits would be inappropriate.
Methods: We propose taking a random sample of SRs published between 2016-2018 from KSR Evidence (a database of SRs) to investigate the use of English-language limits. We intend to analyse SRs which do not apply an English-language limit to see how many include non-English studies. We will investigate if there is a relationship between searching quality of SRs, as assessed by D2, and language limits. We also want to see if there are connections to topics which include non-English studies.
Results: In a pilot study, we analysed a random sample of 168 SRs published in 2016. Ninety-four SRs did not have English-language limits. Twenty-four (25%) of these SRs had at least one non-English include. Fifty-three (56%) SRs which did not apply English-language limits had a low risk of bias in D2. Conversely, 47/74 (64%) of studies which applied an English language limit were judged as high risk of bias in D2. The SRs which included non-English studies were varied in topic and we could not identify a connection to topic and the likelihood of including a non-English study.
Conclusions:Based on preliminary findings, we argue that if 25% of SRs which did not apply an English-language limit, went on to include at least one non-English study, the application of language limits means there is a risk of missing relevant evidence which, in the worst scenario, could change outcomes. At a minimum the inclusion of relevant non-English studies increases sample size and the certainty of outcomes. Preliminary findings suggest that topics which included non-English studies are varied and therefore English-language limits cannot be justified at searching stage. We conclude that SRs which did not apply a language limit are likely to have search methods which are low risk of bias and thus have searched widely with a comprehensive strategy. Findings may change.
Patient or healthcare consumer involvement: None
Objectives: The objective of our research is to see how many SRs which do not use English-language limits, go on to have non-English included studies. We want to investigate connections between an SR which does not impose language restrictions and its risk of bias in Domain 2 (D2) of the ROBIS (Risk of Bias) tool (Identification and Selection of Studies). We also want to see if there are patterns in relation to topic and the inclusion of non-English studies which might indicate when English-language limits would be inappropriate.
Methods: We propose taking a random sample of SRs published between 2016-2018 from KSR Evidence (a database of SRs) to investigate the use of English-language limits. We intend to analyse SRs which do not apply an English-language limit to see how many include non-English studies. We will investigate if there is a relationship between searching quality of SRs, as assessed by D2, and language limits. We also want to see if there are connections to topics which include non-English studies.
Results: In a pilot study, we analysed a random sample of 168 SRs published in 2016. Ninety-four SRs did not have English-language limits. Twenty-four (25%) of these SRs had at least one non-English include. Fifty-three (56%) SRs which did not apply English-language limits had a low risk of bias in D2. Conversely, 47/74 (64%) of studies which applied an English language limit were judged as high risk of bias in D2. The SRs which included non-English studies were varied in topic and we could not identify a connection to topic and the likelihood of including a non-English study.
Conclusions:Based on preliminary findings, we argue that if 25% of SRs which did not apply an English-language limit, went on to include at least one non-English study, the application of language limits means there is a risk of missing relevant evidence which, in the worst scenario, could change outcomes. At a minimum the inclusion of relevant non-English studies increases sample size and the certainty of outcomes. Preliminary findings suggest that topics which included non-English studies are varied and therefore English-language limits cannot be justified at searching stage. We conclude that SRs which did not apply a language limit are likely to have search methods which are low risk of bias and thus have searched widely with a comprehensive strategy. Findings may change.
Patient or healthcare consumer involvement: None