Background: Randomized controlled trials with inadequate allocation concealment or a lack of double-blinding are associated with biased estimates of treatment effects. However, empirical evidence of bias associated with the methodologic attributes of non-randomized studies is lacking.

Objectives: To identify methodologic attributes of non-randomized studies that are associated with biased estimates of treatment effect.

Methods: We identified English language randomized and non-randomized studies comparing laparoscopic- assisted and conventional surgery for colon cancer. We estimated odds ratios of the risk of post-operative complications as the treatment effect of interest. Meta-regression was used to determine the association between systematic outcome assessment— an explicit statement of a standardized protocol for the collection of outcome data— and estimates for treatment effect.

Results: We identified 155 comparative studies from 6,261 abstracts. Of which, 62studies (41NRS and 21RCTs) involving 857,132 patients provided estimates of the risk of post-operative complications. Studies where outcomes were not assessed in a systematic manner estimated larger treatment effects; odds ratios were exaggerated by 26% (95% CI, 4%-44%) as compared with studies with systematic outcome assessment.

Conclusions: Non-randomized studies with systematic outcome assessment are less prone to bias.