Article type
Year
Abstract
Background: The sensitivity of computer strategies varies between 50-80%1 in identifying randomised controlled trials (RCTs). These reports, however, do not take into consideration the quality of the RCTs. Evidence exists that the majority of good quality RCTs lie within computer databases2. We aim to demonstrate this in pain relief research and define the sensitivity and specificity of three computer strategies as compared to existing gold standards.
Methods: Two gold standards using exhaustive hand and computer searches that referenced quality RCTs were selected. 1. Fifteen systematic reviews published by the Oxford University Pain Research Unit contained in their textbook An evidence-based resource for pain relief.3 2. Pain control after thoracic surgery - a review of current techniques4 Department of Anaesthesia, Standford University. Three computer strategies were selected. 1. Optimally Sensitive Search Strategy (OSSS) - a 29-line strategy1, contained in the Cochrane Collaboration Handbook. 2. Randomized controlled trials.publication type (RCTs.pt) - introduced in 1991 by the US National Library of Medicine. 3. (double blind$ or random$).af. - A single-line computer algorithm (abbreviated to DR.af) based on the assumption that double blinded RCTs would be indexed with 'double blind', 'random' or variations of these terms in MEDLINE/EMBASE. Results: The 15 Oxford systematic reviews contained 289 quality RCTs; 280 (97%) of them were listed in MEDLINE or EMBASE. Of the computer-listed RCTs the OSSS was highly sensitive (99.6%), followed by DR.af (96%) and RCTs.pt (67%). The respective sensitivities over the years are presented in figure 1. It is interesting that the sensitivity of DR.af increased progressively, independent of RCTs.pt (between 1971-1980) and is greater than 95% since 1980. Within the 15 reviews, there were 30 analyses that contained four or more RCTs; DR.af was 97% sensitive, ranging from 75-100% with the mode at 100% (18/30). As a result, the RCTs identified by DR.af drew the same conclusion in 97% (29/30) of the analyses. Kavanagh's review4 referenced 108 articles, of which 105 (97%) were computer listed. Sensitivity (%) Kavanagh documented the quality of all the referenced articles: 32 high quality RCTs that fulfilled the inclusion criteria, 38 RCTs that did not and 35 non-RCTs. The specificity (non-RCTs excluded/all non-RCTs) was 54%, 89% and 89% for the OSSS, DR.af and RCTs.pt respectively (Table 1). Table 1 Kavanagh OSSS DR.af RCTs.pt Quality RCTs identified 32 32 32 30 Non-RCTs identified 35 16 4 4 Non-RCTs excluded - 19 31 31 Years Conclusions: In pain relief research, the majority of quality RCTs are contained in computer databases. Furthermore, the OSSS and DR.af were highly sensitive and were representative of the overall clinical picture. The difference between the two strategies lies in their ability to exclude non-RCTs. The OSSS excluded approximately one in two non-RCTs, whereas DR.af excluded nine in ten. We are currently working towards defining the potential of the OSSS and DR.af in other fields. These strategies may translate into substantial savings in time, effort and cost in organising and adapting systematic reviews. Reference List 1. Dickersin K, Scherer R, Lefebvre C. Identifying relevant studies for systematic reviews . BMJ 1994; 309:1286-1291. 2. Knipschild P. Systematic reviews. Some examples. BMJ 1994; 309:719-721. 3. Henry McQuay, Andrew Moore. An evidence-based resource for pain relief. Oxford: Oxford University Press, 1998; 4. Kavanagh BP, Katz J, Sandler AN. Pain control after thoracic surgery. A review of current techniques. Anesthesiology 1994;81:737-759.
Methods: Two gold standards using exhaustive hand and computer searches that referenced quality RCTs were selected. 1. Fifteen systematic reviews published by the Oxford University Pain Research Unit contained in their textbook An evidence-based resource for pain relief.3 2. Pain control after thoracic surgery - a review of current techniques4 Department of Anaesthesia, Standford University. Three computer strategies were selected. 1. Optimally Sensitive Search Strategy (OSSS) - a 29-line strategy1, contained in the Cochrane Collaboration Handbook. 2. Randomized controlled trials.publication type (RCTs.pt) - introduced in 1991 by the US National Library of Medicine. 3. (double blind$ or random$).af. - A single-line computer algorithm (abbreviated to DR.af) based on the assumption that double blinded RCTs would be indexed with 'double blind', 'random' or variations of these terms in MEDLINE/EMBASE. Results: The 15 Oxford systematic reviews contained 289 quality RCTs; 280 (97%) of them were listed in MEDLINE or EMBASE. Of the computer-listed RCTs the OSSS was highly sensitive (99.6%), followed by DR.af (96%) and RCTs.pt (67%). The respective sensitivities over the years are presented in figure 1. It is interesting that the sensitivity of DR.af increased progressively, independent of RCTs.pt (between 1971-1980) and is greater than 95% since 1980. Within the 15 reviews, there were 30 analyses that contained four or more RCTs; DR.af was 97% sensitive, ranging from 75-100% with the mode at 100% (18/30). As a result, the RCTs identified by DR.af drew the same conclusion in 97% (29/30) of the analyses. Kavanagh's review4 referenced 108 articles, of which 105 (97%) were computer listed. Sensitivity (%) Kavanagh documented the quality of all the referenced articles: 32 high quality RCTs that fulfilled the inclusion criteria, 38 RCTs that did not and 35 non-RCTs. The specificity (non-RCTs excluded/all non-RCTs) was 54%, 89% and 89% for the OSSS, DR.af and RCTs.pt respectively (Table 1). Table 1 Kavanagh OSSS DR.af RCTs.pt Quality RCTs identified 32 32 32 30 Non-RCTs identified 35 16 4 4 Non-RCTs excluded - 19 31 31 Years Conclusions: In pain relief research, the majority of quality RCTs are contained in computer databases. Furthermore, the OSSS and DR.af were highly sensitive and were representative of the overall clinical picture. The difference between the two strategies lies in their ability to exclude non-RCTs. The OSSS excluded approximately one in two non-RCTs, whereas DR.af excluded nine in ten. We are currently working towards defining the potential of the OSSS and DR.af in other fields. These strategies may translate into substantial savings in time, effort and cost in organising and adapting systematic reviews. Reference List 1. Dickersin K, Scherer R, Lefebvre C. Identifying relevant studies for systematic reviews . BMJ 1994; 309:1286-1291. 2. Knipschild P. Systematic reviews. Some examples. BMJ 1994; 309:719-721. 3. Henry McQuay, Andrew Moore. An evidence-based resource for pain relief. Oxford: Oxford University Press, 1998; 4. Kavanagh BP, Katz J, Sandler AN. Pain control after thoracic surgery. A review of current techniques. Anesthesiology 1994;81:737-759.