Background: Randomized controlled trials (RCTs) are considered the gold standard for the comparison of treatments. Nevertheless, evidence from non-randomized observational studies (NRSs) is relevant: randomization is sometimes not acceptable to patients, e.g. when treatments differ qualitatively, like surgical vs. medical therapy. For some questions, NRSs may be the sole source of evidence. Objectives: In the analysis of NRSs, a simple overall comparison of the treatment arms may lead to a biased estimate of the treatment effect due to confounding factors. It will be illustrated that the validity of conclusions greatly depends on whether the analysis is adequately adjusted for all important confounders. Methods: We use an example of a study with a so-called Comprehensive Cohort Study (CCS) design, where all patients fulfilling the clinical eligibility criteria and giving consent to participation are recruited. Patients are randomized between study treatments if they give consent to randomization. If not, they are not randomized and receive their preferred study treatment according to the protocol. From the RCT part of the study, an unbiased estimate of the treatment effect will be obtained. From the NRS part of the study, different estimates of the treatment effect will be calculated, unadjusted and adjusted for different sets of covariates und using different methods for adjustment, i.e. conventional regression analysis, and propensity score based adjustment. Results: The size of the treatment effect estimate from the NRS part is greatly dependent on which confounders are adjusted for. This will be contrasted to the unbiased treatment effect estimate from the RCT part. If these were combined in a meta-analysis, the result of the meta-analysis would also greatly depend on the factors being accounted for in the analysis of the NRS part. Conclusions: The CCS design offers an ideal situation for illustrating the effects of different adjustment methods on the resulting effect estimate. Since the correct adjustment for confounders is vital for a valid interpretation of results of NRSs, systematic reviews incorporating data from NRSs should preferentially be based on individual patient data metaanalyses.