Article type
Year
Abstract
Background: Multiple studies suggest that effect estimates from non-randomized studies (NRS) are comparable to those from randomized controlled trials (RCTs). There is also evidence of bias associated with specific design characteristics of RCTs. Accordingly, comparisons of NRS and RCTs to date have often compared NRS with a heterogeneous group of RCTs.
Objectives: To compare the results of NRS with those of RCTs at low risk of bias. Studies evaluating the surgical treatment of colon cancer were used for this case study.
Methods: All studies comparing laparoscopy with conventional surgery for the management of colon cancer were identified. Bayesian meta-analysis was separately performed for two subjective outcomes, post-operative complications (binary) and length of stay (LOS, continuous) and two objective outcomes, mortality (binary) and number of lymph nodes harvested (continuous). Meta-analysis was performed for (i) All Studies, (ii) NRS, (iii) RCTs, (iv) Typical RCTs and (v) Strong RCTs. The Cochrane Risk of Bias Tool was used to classify studies as ‘Strong’ (low risk of bias) or ‘Typical’ (unclear and high risk of bias). A Bayesian meta-regression was conducted with study design (NRS, Typical RCT, Strong RCT) as a predictor variable. Sensitivity analyses assessed the impact of period effects and between-study case-mix.
Results: 145 studies reported the outcomes of interest (Table 1). For post-operative complications, the odds ratios from NRS were 36% smaller (i.e. demonstrating more benefit) than those from Strong RCTs [ROR 0.64, (0.42-0.97), p = 0.04] (Fig. 1). The same exaggerated benefit among NRS was seen when assessing LOS, [Difference in Mean Difference, −2.15, (−4.08, −0.21), p = 0.03]. This pattern was not observed for the objective outcomes (mortality, p = 0.38, and number of LN harvested, p = 0.62). Meta-regression results, adjusted for period effects and case-mix between studies, showed persistent bias among NRS.
Conclusions: When evaluating subjective outcomes, effect estimates from NRS may be associated with significant bias.
Objectives: To compare the results of NRS with those of RCTs at low risk of bias. Studies evaluating the surgical treatment of colon cancer were used for this case study.
Methods: All studies comparing laparoscopy with conventional surgery for the management of colon cancer were identified. Bayesian meta-analysis was separately performed for two subjective outcomes, post-operative complications (binary) and length of stay (LOS, continuous) and two objective outcomes, mortality (binary) and number of lymph nodes harvested (continuous). Meta-analysis was performed for (i) All Studies, (ii) NRS, (iii) RCTs, (iv) Typical RCTs and (v) Strong RCTs. The Cochrane Risk of Bias Tool was used to classify studies as ‘Strong’ (low risk of bias) or ‘Typical’ (unclear and high risk of bias). A Bayesian meta-regression was conducted with study design (NRS, Typical RCT, Strong RCT) as a predictor variable. Sensitivity analyses assessed the impact of period effects and between-study case-mix.
Results: 145 studies reported the outcomes of interest (Table 1). For post-operative complications, the odds ratios from NRS were 36% smaller (i.e. demonstrating more benefit) than those from Strong RCTs [ROR 0.64, (0.42-0.97), p = 0.04] (Fig. 1). The same exaggerated benefit among NRS was seen when assessing LOS, [Difference in Mean Difference, −2.15, (−4.08, −0.21), p = 0.03]. This pattern was not observed for the objective outcomes (mortality, p = 0.38, and number of LN harvested, p = 0.62). Meta-regression results, adjusted for period effects and case-mix between studies, showed persistent bias among NRS.
Conclusions: When evaluating subjective outcomes, effect estimates from NRS may be associated with significant bias.
