Article type
Year
Abstract
Background: Handbooks advise systematic reviewers to check reference lists (CRL) of primary and/or secondary studies to identify additional material for systematic reviews. Evidence-based practice would weigh the sensitivity and specificity of this approach against the resource implications and the evidence regarding the potential to protect against or introduce bias into the review.
Objectives: To systematically review the research evidence regarding CRL, examining these key questions: What is the marginal yield of CRL? Is it most productive to focus the CRL search on certain publication types? Does CRL a) help protect the review against bias? or b) potentially introduce bias? What is the evidence regarding cost-benefit of CRL? What is the utility of using reference lists for developing electronic searches, or validating their recall?
Methods: We searched The Cochrane Methodology Register, Medline, Library Literature and LISA to identify published or unpublished primary research examining CRL. Case reports of the yield from CRL in a single systematic review were excluded. We also determined the prevalence of CRL in a sample of 105 Cochrane reviews.
Results: CRL was reported by 86.7% of Cochrane reviews and was the most common search technique, after searching electronic databases. Initial screening has identified three eligible studies. Preliminary findings suggest CRL may identify novel unpublished material. [1, 2] One study observed positive bias within reference lists (citation bias). [3]
Discussion: Recommendations to check reference lists are not evidence-based. The practice has not been studied systematically and evidence is largely from expert opinion and case reports (Grade D). [4]
References: 1 Vickers AJ, Smith C. Incorporating data from dissertations in systematic reviews. Int J Technol Assess Health Care 2000; 16(2):711-713.
2. Royle P, Waugh N. Literature searching for clinical and cost-effectiveness studies used in health technology assessment reports carried out for the National Institute for Clinical Excellence appraisal system. Health Technol Assess 2003; 7(34):1-64.
3. Gotzsche PC. Reference bias in reports of drug trials. Br Med J (Clin Res Ed) 1987; 295(6599):654-656.
4. Phillips B, Ball C, Sackett D, Badenoch D, Straus S, Haynes RB, and Dawes M. Oxford Centre for Evidence-based Medicine Levels of Evidence. Available at: http://www.cebm.net/levels_of_evidence.asp.
Objectives: To systematically review the research evidence regarding CRL, examining these key questions: What is the marginal yield of CRL? Is it most productive to focus the CRL search on certain publication types? Does CRL a) help protect the review against bias? or b) potentially introduce bias? What is the evidence regarding cost-benefit of CRL? What is the utility of using reference lists for developing electronic searches, or validating their recall?
Methods: We searched The Cochrane Methodology Register, Medline, Library Literature and LISA to identify published or unpublished primary research examining CRL. Case reports of the yield from CRL in a single systematic review were excluded. We also determined the prevalence of CRL in a sample of 105 Cochrane reviews.
Results: CRL was reported by 86.7% of Cochrane reviews and was the most common search technique, after searching electronic databases. Initial screening has identified three eligible studies. Preliminary findings suggest CRL may identify novel unpublished material. [1, 2] One study observed positive bias within reference lists (citation bias). [3]
Discussion: Recommendations to check reference lists are not evidence-based. The practice has not been studied systematically and evidence is largely from expert opinion and case reports (Grade D). [4]
References: 1 Vickers AJ, Smith C. Incorporating data from dissertations in systematic reviews. Int J Technol Assess Health Care 2000; 16(2):711-713.
2. Royle P, Waugh N. Literature searching for clinical and cost-effectiveness studies used in health technology assessment reports carried out for the National Institute for Clinical Excellence appraisal system. Health Technol Assess 2003; 7(34):1-64.
3. Gotzsche PC. Reference bias in reports of drug trials. Br Med J (Clin Res Ed) 1987; 295(6599):654-656.
4. Phillips B, Ball C, Sackett D, Badenoch D, Straus S, Haynes RB, and Dawes M. Oxford Centre for Evidence-based Medicine Levels of Evidence. Available at: http://www.cebm.net/levels_of_evidence.asp.