Article type
Year
Abstract
Background: Discrepancy between reported methodological quality and the actual methodological quality of randomized controlled trials (RCTs) has been documented. However, the relative impact of reported versus actual methodological quality on the treatment effect size (ES) has not been assessed.
Objective: To assess the impact of reported methodological quality versus actual methodological quality on ES.
Methods: All consecutive terminated phase III RCTs published by 8 National Cancer Institute sponsored Cooperative Groups until year 2006 were eligible for inclusion. Data on methodological quality domains relevant to minimizing bias and random error were extracted from protocols and publications for each study. 'Actual quality’ was assessed based on data from either protocols or the publications. The hazard ratio (HR) for overall survival was used as the ES. Association between methodological quality (reported vs. actual) with ES was conducted using standard meta-epidemiologic methods.
Results: A total of 429 RCTs met the inclusion criteria. There was no statistically significant difference between associations of ES and actual vs. methodological quality for the following domains: adequacy of randomization sequence generation, description of drop outs, intention to treat analysis, pre specification of alpha and beta errors (Figure1). However, on average, poorly reported allocation concealment exaggerated the ES by 6% (Ratio of HRs: 0.94, 95% CI: 0.88-0.99). Also, poorly reported blinding inflated the ES by 24% (Ratio of HRs: 1.24, 95% CI: 1.05-1.43). Nonetheless, in the 'actual quality’ assessment, no significant association between ES and any of the methodological quality domains was detected.
Conclusion: Our study results show that the assessment of the impact of quality on the ES based on the quality of reporting only can produce misleading results. These findings are important for users of research evidence, systematic reviewers, meta-epidemiologic research, as well as further development of the Cochrane 'risk of bias' table which relies on reported methodological quality.
Objective: To assess the impact of reported methodological quality versus actual methodological quality on ES.
Methods: All consecutive terminated phase III RCTs published by 8 National Cancer Institute sponsored Cooperative Groups until year 2006 were eligible for inclusion. Data on methodological quality domains relevant to minimizing bias and random error were extracted from protocols and publications for each study. 'Actual quality’ was assessed based on data from either protocols or the publications. The hazard ratio (HR) for overall survival was used as the ES. Association between methodological quality (reported vs. actual) with ES was conducted using standard meta-epidemiologic methods.
Results: A total of 429 RCTs met the inclusion criteria. There was no statistically significant difference between associations of ES and actual vs. methodological quality for the following domains: adequacy of randomization sequence generation, description of drop outs, intention to treat analysis, pre specification of alpha and beta errors (Figure1). However, on average, poorly reported allocation concealment exaggerated the ES by 6% (Ratio of HRs: 0.94, 95% CI: 0.88-0.99). Also, poorly reported blinding inflated the ES by 24% (Ratio of HRs: 1.24, 95% CI: 1.05-1.43). Nonetheless, in the 'actual quality’ assessment, no significant association between ES and any of the methodological quality domains was detected.
Conclusion: Our study results show that the assessment of the impact of quality on the ES based on the quality of reporting only can produce misleading results. These findings are important for users of research evidence, systematic reviewers, meta-epidemiologic research, as well as further development of the Cochrane 'risk of bias' table which relies on reported methodological quality.
Images
