Article type
Year
Abstract
Background: Randomized clinical trials (RCT) remain the best way for investigating the effects of new interventions for the prevention and cure of cancer. However, if a RCT fails to show a significant difference between the experimental and the control intervention, is it because the new intervention truly is not effective (evidence of absence of treatment effect) or because the trial's results were inconclusive (absence of evidence of treatment effect)?
Methods: We extracted data from the original publications and protocols associated with three NCI sponsored Cooperative Groups (Radiation Therapy Oncology Group, Children's Oncology Group and Gynecologic Oncology Group), on 261 trials with a total of more than 50,000 patients. Quality of trials was high. To determine whether non-significant trial findings were true negatives or inconclusive, we compared the a priori effect size used in the power calculation for the trial's pre-defined primary outcome to the confidence intervals (CIs) of the non-significant trial results. Trial results were construed as true negative if the effect size and the 95%CIs were entirely outside the limits of equivalence, and inconclusive if the 95%CIs crossed the line of no effect and one or both limits of equivalence.
Results: 65% of the studies had a non-significant result (169/261). Data on primary outcomes were available for 154 out of 169 studies (91%). 67% of trial (104/154) results were true negative (evidence of absence of treatment effect) and 33% (50/154) were inconclusive (absence of evidence of treatment effect). 73% (113/154) of the studies had undertaken a pre-trial power analysis. A number of trials choose to detect unrealistically large treatment effect size (see graph).
Conclusion: Even high-quality RCTs conducted by prestigious institutions and well respected research groups often produce inconclusive or negative findings (i.e. results that are statistically consistent with both, absence and presence of a benefit that was thought to be clinically important when the trial was designed). Unrealistic expectations in treatment effect size appear to be a major culprit for continuing conduct of underpowered trials.
This study was supported by US NIH/ORI grant.
Methods: We extracted data from the original publications and protocols associated with three NCI sponsored Cooperative Groups (Radiation Therapy Oncology Group, Children's Oncology Group and Gynecologic Oncology Group), on 261 trials with a total of more than 50,000 patients. Quality of trials was high. To determine whether non-significant trial findings were true negatives or inconclusive, we compared the a priori effect size used in the power calculation for the trial's pre-defined primary outcome to the confidence intervals (CIs) of the non-significant trial results. Trial results were construed as true negative if the effect size and the 95%CIs were entirely outside the limits of equivalence, and inconclusive if the 95%CIs crossed the line of no effect and one or both limits of equivalence.
Results: 65% of the studies had a non-significant result (169/261). Data on primary outcomes were available for 154 out of 169 studies (91%). 67% of trial (104/154) results were true negative (evidence of absence of treatment effect) and 33% (50/154) were inconclusive (absence of evidence of treatment effect). 73% (113/154) of the studies had undertaken a pre-trial power analysis. A number of trials choose to detect unrealistically large treatment effect size (see graph).
Conclusion: Even high-quality RCTs conducted by prestigious institutions and well respected research groups often produce inconclusive or negative findings (i.e. results that are statistically consistent with both, absence and presence of a benefit that was thought to be clinically important when the trial was designed). Unrealistic expectations in treatment effect size appear to be a major culprit for continuing conduct of underpowered trials.
This study was supported by US NIH/ORI grant.
PDF