Background:

A number of recent studies revealed selective reporting of results from randomized trials at the level of endpoints within published studies (outcome reporting bias) or at the level of entire trials being selectively published (study publication bias).

Objectives:

The present study aims (1) to elucidate the extent to which under-reporting of outcomes of primary interest affects Cochrane reviews, and (2) to examine if this under-reporting is biased, i.e. if the smaller the proportion of trials reporting the primary outcome, the greater the reported effect size in favour of the intervention.

Methods:

Of the randomly selected 500 completed systematic reviews from The Cochrane Library, two reviewers independently
identified those which included 10 or more randomized controlled trials (RCTs). For each of these reviews, three reviewers then jointly decided which comparison to focus on and which outcome of that comparison to extract data from, based on both the review authors' description and our own medical knowledge. We extracted the number of RCTs which made that comparison, the number of RCTs which actually contributed data to that comparison-outcome, and the pooled odds ratio (OR) and standardised mean difference (SMD).

Results:

(1) Between 2 and 100 % (mean = 49%) of the identified RCTs contributed to the primary outcomes of interest for the completed reviews. (2) The correlations between the proportion of reporting trials and the effect sizes in terms of OR and SMD were in the hypothesized direction at -0.15 (p=0.03) and -0.18 (p=0.04).

Conclusions:

Even when study publication bias is overcome, under-reporting of primary outcomes is so extensive as to question the validity of many of the completed Cochrane reviews fundamentally. Moreover, this under-reporting appears to be significantly biased; the less often a particular outcome of interest is reported among the identified trials, the greater the pooled effect size.