Article type
Year
Abstract
Background: There has been much work on the development of checklists and scales for validating aspects of the systematic review process. However, a validated process for evaluating the search strategy, which determines the quality and completeness of the evidence base for systematic reviews, does not exist. The lack of such a process paired with a demonstrable level of error in reported searches leaves this type of research open to debate over the quality of evidence on which the review is based.
Objectives: To find and evaluate the evidence concerning a preliminary list of 13 aspects of electronic literatures searches to determine whether there is research evidence to support including that element in a formal search evaluation tool.
Methods: We searched the The Cochrane Methodology Register LISA and MEDLINE to identify published or unpublished research relevant to any of the elements under consideration. Two reviewers screened for relevance and extracted data. Disagreements were resolved through discussion.
Results: For each element, we considered whether there was research evidence concerning any of these aspects of search performance: recall/sensitivity, specificity, precision, reproducibility, and cross-database stability. Elements assessed were spelling mistakes, missed spelling variants, missed index terms, errors in the use of truncation and logical operators, mis-specified line numbers, redundancies, and suitability of the search for use in databases other than MEDLINE.
Discussion: Having summarized existing evidence regarding proposed checklist items, our next steps are to involve stakeholders in the development and validation of this checklist through a consensus-building forum. From this forum we will develop the final list of elements and criteria and guidelines for rating the items on the checklist. These will then be tested based on a test collection of published search strategies from HTA, the Cochrane Collaboration and specific journal articles. This process will be refined and retested with a senior librarian and less experienced searchers.
Objectives: To find and evaluate the evidence concerning a preliminary list of 13 aspects of electronic literatures searches to determine whether there is research evidence to support including that element in a formal search evaluation tool.
Methods: We searched the The Cochrane Methodology Register LISA and MEDLINE to identify published or unpublished research relevant to any of the elements under consideration. Two reviewers screened for relevance and extracted data. Disagreements were resolved through discussion.
Results: For each element, we considered whether there was research evidence concerning any of these aspects of search performance: recall/sensitivity, specificity, precision, reproducibility, and cross-database stability. Elements assessed were spelling mistakes, missed spelling variants, missed index terms, errors in the use of truncation and logical operators, mis-specified line numbers, redundancies, and suitability of the search for use in databases other than MEDLINE.
Discussion: Having summarized existing evidence regarding proposed checklist items, our next steps are to involve stakeholders in the development and validation of this checklist through a consensus-building forum. From this forum we will develop the final list of elements and criteria and guidelines for rating the items on the checklist. These will then be tested based on a test collection of published search strategies from HTA, the Cochrane Collaboration and specific journal articles. This process will be refined and retested with a senior librarian and less experienced searchers.