Article type
Year
Abstract
Background: While selective publication of entire studies has received much attention in the literature, the selective reporting of outcomes within published trials has not undergone significant empirical investigation. Both types of reporting biases may threaten the reliability of the literature upon which systematic reviews are based.
Objective: To evaluate the reporting of outcomes in randomised trials funded by a national government agency.
Methods: The study cohort consisted of all randomised trials whose protocols were approved for funding by the Canadian Institutes of Health Research from 1990-98. Subsequent publications were identified by contacting trialists and performing literature searches of PubMed, EMBASE, and the Cochrane Controlled Trials Register in January 2003. Data on reported and unreported outcomes were recorded from trial protocols, publications and a pre-piloted survey of authors. An outcome was considered incompletely reported if insufficient data were presented for meta-analysis. Odds ratios relating the completeness of outcome reporting to statistical significance were calculated for each trial, and then pooled using a random effects meta-analysis. Finally, we recorded any major discrepancies between primary outcomes specified in protocols and those defined in publications.
Results: 48 published trials (46% of all funded trials) with 68 publications and 1,402 outcomes were identified. 44% of the trials were published in general medical journals. The median sample size was 299 (inner-80-percentile range 61-2568). 94% of trialists responded to the survey with information about unreported outcomes. 31% (inner-80-percentile range 5-67%) of efficacy and 59% (0-100%) of safety outcomes per trial were, on average, incompletely reported for meta-analysis. Statistically significant outcomes had a higher odds of being fully reported compared to non-significant outcomes (odds ratio 2.7, 95% CI 1.5 to 5.0 for efficacy outcomes; 7.7, 0.53 to 111 for safety outcomes). In comparing protocols to publications, 40% of trials had major discrepancies in the specification of primary outcomes. None of the publications for these trials mentioned that amendments had been made to the protocols.
Conclusions: In our sample of relatively large, government-funded trials, incomplete reporting of trial outcomes was common and was associated with a lack of statistical significance. Primary outcomes were often changed, omitted, or newly-introduced in publications relative to protocols. The trial literature may therefore be unreliable, and meta-analyses may tend to produce inflated estimates of treatment effect. To deter the selective reporting of results, protocol details should be made public prior to trial completion.
Objective: To evaluate the reporting of outcomes in randomised trials funded by a national government agency.
Methods: The study cohort consisted of all randomised trials whose protocols were approved for funding by the Canadian Institutes of Health Research from 1990-98. Subsequent publications were identified by contacting trialists and performing literature searches of PubMed, EMBASE, and the Cochrane Controlled Trials Register in January 2003. Data on reported and unreported outcomes were recorded from trial protocols, publications and a pre-piloted survey of authors. An outcome was considered incompletely reported if insufficient data were presented for meta-analysis. Odds ratios relating the completeness of outcome reporting to statistical significance were calculated for each trial, and then pooled using a random effects meta-analysis. Finally, we recorded any major discrepancies between primary outcomes specified in protocols and those defined in publications.
Results: 48 published trials (46% of all funded trials) with 68 publications and 1,402 outcomes were identified. 44% of the trials were published in general medical journals. The median sample size was 299 (inner-80-percentile range 61-2568). 94% of trialists responded to the survey with information about unreported outcomes. 31% (inner-80-percentile range 5-67%) of efficacy and 59% (0-100%) of safety outcomes per trial were, on average, incompletely reported for meta-analysis. Statistically significant outcomes had a higher odds of being fully reported compared to non-significant outcomes (odds ratio 2.7, 95% CI 1.5 to 5.0 for efficacy outcomes; 7.7, 0.53 to 111 for safety outcomes). In comparing protocols to publications, 40% of trials had major discrepancies in the specification of primary outcomes. None of the publications for these trials mentioned that amendments had been made to the protocols.
Conclusions: In our sample of relatively large, government-funded trials, incomplete reporting of trial outcomes was common and was associated with a lack of statistical significance. Primary outcomes were often changed, omitted, or newly-introduced in publications relative to protocols. The trial literature may therefore be unreliable, and meta-analyses may tend to produce inflated estimates of treatment effect. To deter the selective reporting of results, protocol details should be made public prior to trial completion.