Article type
Year
Abstract
Background: The Cochrane HIV/AIDS Specialized Register has been housed at the South African Cochrane Centre since 2004. Records from quarterly searches are coded manually and include a term for randomized controlled trials (RCTs). In this study, we test the accuracy of our Register in identifying RCTs for Cochrane reviews.
Objectives: 1. Determine the precision and sensitivity of the Register compared to electronic searches for identifying RCTs for Cochrane reviews; 2. Identify limitations of the Register.
Methods: We searched Archie to identify Cochrane HIV/AIDS reviews published in 2010. We extracted relevant keywords and used them to search our Register for RCTs relevant in each review. We calculated precision by comparing the number of RCTs identified with the number of records retrieved from the searches, and sensitivity by comparing the number of relevant RCTs identified with the number of RCTs included in each review.
Results: We identified 18 reviews, and have analyzed three to date. For one review, four RCTs were identified from 103 records retrieved from standard electronic databases (precision: 3.9%). Searching the Register found all four RCTs from 10 records, and precision improved to 40%, with no loss of sensitivity. However, for the other two reviews, with precision increasing from 0.6% (13/2317) to 9.5% (8/84) when searching the Register and from 5.6% (34/640) to 24.2% (32/132), with 38% (5/13) and 6% (2/34) loss in sensitivity, respectively. Reasons for loss of sensitivity included keywords not appearing in the title or abstract of the Register record (n = 1); records coded as awaiting assessment (n=3); records not identified in the Register (n=3).
Conclusions: Searching the Register for HIV/AIDS trials greatly increased precision but some sensitivity was lost with reasons identified. The Register is not yet sufficiently robust to be the sole database searched and additional database searches are required to maintain review break quality.
Objectives: 1. Determine the precision and sensitivity of the Register compared to electronic searches for identifying RCTs for Cochrane reviews; 2. Identify limitations of the Register.
Methods: We searched Archie to identify Cochrane HIV/AIDS reviews published in 2010. We extracted relevant keywords and used them to search our Register for RCTs relevant in each review. We calculated precision by comparing the number of RCTs identified with the number of records retrieved from the searches, and sensitivity by comparing the number of relevant RCTs identified with the number of RCTs included in each review.
Results: We identified 18 reviews, and have analyzed three to date. For one review, four RCTs were identified from 103 records retrieved from standard electronic databases (precision: 3.9%). Searching the Register found all four RCTs from 10 records, and precision improved to 40%, with no loss of sensitivity. However, for the other two reviews, with precision increasing from 0.6% (13/2317) to 9.5% (8/84) when searching the Register and from 5.6% (34/640) to 24.2% (32/132), with 38% (5/13) and 6% (2/34) loss in sensitivity, respectively. Reasons for loss of sensitivity included keywords not appearing in the title or abstract of the Register record (n = 1); records coded as awaiting assessment (n=3); records not identified in the Register (n=3).
Conclusions: Searching the Register for HIV/AIDS trials greatly increased precision but some sensitivity was lost with reasons identified. The Register is not yet sufficiently robust to be the sole database searched and additional database searches are required to maintain review break quality.