Article type
Year
Abstract
Background: Routinely collected data (RCD) studies are proposed to inform health care decisions when randomized controlled trials (RCTs) are not timely available, but have an inherent risk of selection bias due to confounding by indication. Propensity-score-analyses are frequently used to address selection bias issues, but biases due to remaining unaccounted confounders cannot be excluded.
Objectives: To assess the agreement between results of RCD-studies and subsequent RCTs on the same topic in areas without RCT evidence to guide health care decisions.
Methods: We searched PubMed (up to November 2014) for RCD-studies published up to 2010 that used propensity scores and reported comparative effects of medical interventions for mortality endpoints. We included RCD-studies that were conducted before any RCT was published for the same topic and same compared interventions. We searched systematically for subsequently published pertinent RCTs, extracted mortality data, and calculated the mortality odds ratio (OR). When more than one subsequent RCT was identified, we combined them with random-effects models (or Peto’s approach for rare events) and calculated the summary OR. We analyzed the agreement between RCD-studies and RCTs by calculating the relative OR (ROR; i.e. summary OR of RCT(s) divided by RCD-study estimate; ROR > 1 means RCD overestimate survival benefits). We then synthesized the individual ROR data across all RCD-RCT pairs to calculate the summary ROR (sROR) as overarching measure of the agreement of early RCD-effects and subsequent RCT-results.
Results: We identified 929 records, evaluated 420 in full-text, and screened 124 RCD-studies further. We analyzed 16 RCD-studies without preceding RCTs and for which subsequent pertinent RCTs were identified. Preliminary results indicate limited agreement between RCD-effects and subsequent RCT evidence. RCD-studies showed significantly inflated results (sROR 1.31; 95% CI 1.04 to 1.65).
Conclusions: RCD-studies that have been conducted prior to RCTs seem to systematically and substantially overestimate the benefits of medical treatments. Final results will be available at the Cochrane Colloquium 2015.
Objectives: To assess the agreement between results of RCD-studies and subsequent RCTs on the same topic in areas without RCT evidence to guide health care decisions.
Methods: We searched PubMed (up to November 2014) for RCD-studies published up to 2010 that used propensity scores and reported comparative effects of medical interventions for mortality endpoints. We included RCD-studies that were conducted before any RCT was published for the same topic and same compared interventions. We searched systematically for subsequently published pertinent RCTs, extracted mortality data, and calculated the mortality odds ratio (OR). When more than one subsequent RCT was identified, we combined them with random-effects models (or Peto’s approach for rare events) and calculated the summary OR. We analyzed the agreement between RCD-studies and RCTs by calculating the relative OR (ROR; i.e. summary OR of RCT(s) divided by RCD-study estimate; ROR > 1 means RCD overestimate survival benefits). We then synthesized the individual ROR data across all RCD-RCT pairs to calculate the summary ROR (sROR) as overarching measure of the agreement of early RCD-effects and subsequent RCT-results.
Results: We identified 929 records, evaluated 420 in full-text, and screened 124 RCD-studies further. We analyzed 16 RCD-studies without preceding RCTs and for which subsequent pertinent RCTs were identified. Preliminary results indicate limited agreement between RCD-effects and subsequent RCT evidence. RCD-studies showed significantly inflated results (sROR 1.31; 95% CI 1.04 to 1.65).
Conclusions: RCD-studies that have been conducted prior to RCTs seem to systematically and substantially overestimate the benefits of medical treatments. Final results will be available at the Cochrane Colloquium 2015.