Article type
Year
Abstract
Background: Typical Cochrane systematic reviews limit inclusion criteria to randomized controlled trials (RCTs). More recently, some authors have proposed that systematic reviews of cohort studies actually yield similar results to systematic reviews of randomized controlled trials. Current published papers discuss reliability of the reporting of systematic reviews and the effects of different inclusion criteria on the overall summary statistic. However, validity requires the comparison of different outcomes of a measure to a gold standard.
Objectives: To describe the methodology of a study that will compare the predictions of different review approaches to a "reference standard" that is based on years of subsequent study.
Methods: We are using an historical cohort design. Three topics have been chosen for which there are many RCTs already published. For each condition, we will perform an exhaustive literature search and categorize the articles temporally, and methodologically (RCT, cohort, other observational, basic science). Next, we will identify the point in time when 1 RCT has been published. One reviewer will abstract the data for the RCT, and for all other studies published at that point in time. A second reviewer will verify the data abstraction. The data abstraction and original articles will be distributed among 5 different persons with experience in performing systematic reviews. All 5 will provide interpretations based on the RCT evidence alone, 3 will provide interpretations on the RCT + cohort evidence alone, 2 for the RCT + cohort + other observational evidence alone, and 1 will interpret all human and basic science evidence. We will repeat this process at the point in time that 3, 5, 10 and 20 RCTs have been published.
Results: The interpretations at each time point will be compared to the results of studies that have been published after that date. In addition, we will assess reliability of interpretation based on identical data abstractions at each time point i.e. there are 5 different persons interpreting the RCT evidence at each time point, 3 interpreting the cohort evidence, and 2 interpreting all the observational evidence.
Conclusions: The study is currently in progress.
Objectives: To describe the methodology of a study that will compare the predictions of different review approaches to a "reference standard" that is based on years of subsequent study.
Methods: We are using an historical cohort design. Three topics have been chosen for which there are many RCTs already published. For each condition, we will perform an exhaustive literature search and categorize the articles temporally, and methodologically (RCT, cohort, other observational, basic science). Next, we will identify the point in time when 1 RCT has been published. One reviewer will abstract the data for the RCT, and for all other studies published at that point in time. A second reviewer will verify the data abstraction. The data abstraction and original articles will be distributed among 5 different persons with experience in performing systematic reviews. All 5 will provide interpretations based on the RCT evidence alone, 3 will provide interpretations on the RCT + cohort evidence alone, 2 for the RCT + cohort + other observational evidence alone, and 1 will interpret all human and basic science evidence. We will repeat this process at the point in time that 3, 5, 10 and 20 RCTs have been published.
Results: The interpretations at each time point will be compared to the results of studies that have been published after that date. In addition, we will assess reliability of interpretation based on identical data abstractions at each time point i.e. there are 5 different persons interpreting the RCT evidence at each time point, 3 interpreting the cohort evidence, and 2 interpreting all the observational evidence.
Conclusions: The study is currently in progress.