Developing efficient search strategies to identify papers on adverse events. A: testing and precision sensitivity

Tags: Oral
Golder S, Duffy S, Glanville J, McIntosh H, Miles J

Background: Recent research by White et al [1] has used word frequency and statistical analysis to achieve objectively derived search filters for retrieving systematic reviews. Little research has been conducted on creating search filters to retrieve reports of adverse events. White et al's methods were used in this study to find the most efficient search terms (in terms of high sensitivity with acceptable precision) to retrieve papers reporting adverse events from MEDLINE and EMBASE.

Objectives: To find the most efficient search terms to identify reports of adverse events in MEDLINE and EMBASE.

Methods: A systematic review of the effectiveness and adverse effects of seven new anti-epileptic drugs was used as a case study. This review included a thorough search for studies of effectiveness and additional searches for adverse events. The adverse events papers were found by searching MEDLINE, and EMBASE using a combination of 5 different search approaches, as well as a search of TOXLINE, contacting experts, checking bibliographies, industry submissions and the results of effectiveness searches on the topic. 82 papers included in a systematic review of adverse events of seven new anti-epileptic drugs were used as a quasi gold standard. A comparison set of records for use in the statistical analysis was compiled by randomly selecting papers that failed to meet the review inclusion criteria.

Frequently occurring words within the quasi gold standard set were identified using SIMSTAT for Windows. The occurrence of those words in the quasi gold standard and the comparison sets of records were recorded. The data were analysed, using logistic regression in SPSS for Windows, to determine which words were best at discriminating adverse events records from other records (which had also been retrieved when looking for adverse events).

Results: Preliminary results suggest that there are no clearly discriminating search terms to identify adverse events records in this topic. Further statistical analysis is being conducted, varying the volume of records in the comparison group and word frequency cut off levels, to verify if these preliminary results are maintained at a lower statistical power.

Conclusions: Our results from this case study indicate that searching for reports of adverse events is complex, and that different approaches and combinations of approaches may be required. More efficient retrieval could be enhanced by improvements to indexing practice by either more thorough use of floating subheadings or the allocation of Publication Types or specific indexing terms to connotate reports of adverse events.

References: 1. White VJ, Glanville JM, Lefebvre C, Sheldon TA. A statistical approach to designing search filters to find systematic reviews: objectivity enhances accuracy. J Info Sci. 2001;27:357-70. 2. Glanville JM, Lefebvre C, Camosso-Stefinovic J, Miles J. Finding RCTs becomes easier. To be presented at HTAi May 30 - June 2 2004, Krakow, Poland.