Article type
Year
Abstract
Locating relevant study reports for a research synthesis/meta-analysis can be conceptualized as a multistage process. The first stage involves the use of multiple search strategies to identify the citations of most or all studies relevant to the question under study. The second stage involves the review of identified citations with relevance-based judgments as to their retrieval. The third involves review of retrieved documents with judgments as to their inclusion in the review, with study reports meeting the review's inclusion/exclusion criteria termed source studies. The focus here is on the Potential Source Study Classification System, a systematic method of managing the second stage of this process.
The broad and exhaustive searches typically involved in research syntheses, since most reviewers have the intention of identifying most or all of the relevant literature, tend to result in extremely large numbers of citations, many of which are not reports of empirical research. Retrieval of the documents associated with all identified citations would be prohibitively expensive, particularly since only some of these citations are reports of relevant research. Not retrieving possibly relevant citations increases the risk that the studies reviewed represent a biased sample of research on the topic under review. The difficulty lies in maximizing the yield of relevant documents while minimizing that of irrelevant ones.
The Potential Source Study Classification System was developed to achieve this goal. It is used after identifying: 1) the problem/topic under study, 2) inclusion and exclusion criteria, 3) search strategies, including relevant disciplines, databases, and key terms. During literature searches relevant information that may be helpful to the retrieval decision, such as database abstracts and descriptors, are requested in addition to the citations themselves. The PSSCS involves: 1) retrieving maximal information from the source (database, article, person, etc.), 2) reviewing all information about each citation, and 3) classifying each citation as to its relevance. Each citation and accompanying information is then reviewed and categorized as: 1) a potential source study (PSS), 2) possibly a potential source study (PPSS), or 3) not a potential source study (NPSS). A priori retrieval decisions are associated with each category.
The PSSCS was used in a meta-analysis on the effectiveness of stress coping interventions. A total of 5,683 citations identified from computer searches of 3 reference databases (PsyINFO, ERIC, and Medline) were reviewed and categorized using the PSSCS with the following results: 195 PSS, 171 PPSS, and 5317 NPSS. Retrieval of all PSS citations and a convenience sample of PPSS citations was attempted. All PSS citations (195) and 41/171 PPSS citations were retrieved. Forty of the 236 documents obtained and reviewed from the identified citations met all the inclusion criteria and were included in the meta-analysis. All 40 studies were categorized PSS; one was also categorized PPSS. The remaining 182 did not meet at least one of the inclusion criteria. It is important to note that the only inclusion criteria not met by 88 PSS citations comprising 74 studies and 7 PPSS citations comprising 3 studies was provision of sufficient information for effect size estimation. Use of the PSSCS provided a systematic and explicit basis for retrieval decisions.
The broad and exhaustive searches typically involved in research syntheses, since most reviewers have the intention of identifying most or all of the relevant literature, tend to result in extremely large numbers of citations, many of which are not reports of empirical research. Retrieval of the documents associated with all identified citations would be prohibitively expensive, particularly since only some of these citations are reports of relevant research. Not retrieving possibly relevant citations increases the risk that the studies reviewed represent a biased sample of research on the topic under review. The difficulty lies in maximizing the yield of relevant documents while minimizing that of irrelevant ones.
The Potential Source Study Classification System was developed to achieve this goal. It is used after identifying: 1) the problem/topic under study, 2) inclusion and exclusion criteria, 3) search strategies, including relevant disciplines, databases, and key terms. During literature searches relevant information that may be helpful to the retrieval decision, such as database abstracts and descriptors, are requested in addition to the citations themselves. The PSSCS involves: 1) retrieving maximal information from the source (database, article, person, etc.), 2) reviewing all information about each citation, and 3) classifying each citation as to its relevance. Each citation and accompanying information is then reviewed and categorized as: 1) a potential source study (PSS), 2) possibly a potential source study (PPSS), or 3) not a potential source study (NPSS). A priori retrieval decisions are associated with each category.
The PSSCS was used in a meta-analysis on the effectiveness of stress coping interventions. A total of 5,683 citations identified from computer searches of 3 reference databases (PsyINFO, ERIC, and Medline) were reviewed and categorized using the PSSCS with the following results: 195 PSS, 171 PPSS, and 5317 NPSS. Retrieval of all PSS citations and a convenience sample of PPSS citations was attempted. All PSS citations (195) and 41/171 PPSS citations were retrieved. Forty of the 236 documents obtained and reviewed from the identified citations met all the inclusion criteria and were included in the meta-analysis. All 40 studies were categorized PSS; one was also categorized PPSS. The remaining 182 did not meet at least one of the inclusion criteria. It is important to note that the only inclusion criteria not met by 88 PSS citations comprising 74 studies and 7 PPSS citations comprising 3 studies was provision of sufficient information for effect size estimation. Use of the PSSCS provided a systematic and explicit basis for retrieval decisions.