Article type
Abstract
Background: A fundamental principle of evidence-based medicine (EBM) is that high-quality (certainty) evidence (CoE) provides trustworthy estimates of effects. However, the behavior of the public during the last several years that resulted in the promotion of health interventions based on the lower CoE has questioned the scientific authenticity of this fundamental EBM principle. Therefore, knowing the probability that treatment effects obtained from non-high CoE are "true" is of paramount public interest.
Objectives: To determine the probability of "true" effect estimates in different CoE levels.
Methods: We reasoned that stable treatment effect estimates over time indicate truthfulness. We compared odds ratios (ORs) from meta-analyses (MAs) before and after updates, expecting higher CoE to show similar ORs yielding a ratio of odds ratios (ROR) equal to 1. ROR=1 was predicted to be more common in higher CoE MAs. We analyzed 150 Cochrane MA pairs, using previously detailed methods (doi: 10.1111/jep.13657) to see if ROR = 1 (+/- a deviation margin) aligned with CoE levels according to GRADE methods. These MAs more likely represent “true” effects.
Results: Figure 1 shows a linear relationship between the probability of a true estimate of treatment effects as a function of CoE (at 10% margin error in ROR estimates) (R2=0.99; p=0.042) The probability of potentially "true" estimates decreases by about 20% (95%CI: 15 to 26%) for each drop in the rating of CoE. A linear relationship was less clear with a 5% ROR error margin, likely due to smaller sample size. Still, higher CoE showed significantly greater likelihood of true effects (53%) compared to non-high (i.e., moderate, low, or very low) CoE (25%); p=0.021.
Conclusion: This study offers the first estimate of potentially true treatment effects based on CoE. These findings can help refine decision-making in policy and clinical settings.
Objectives: To determine the probability of "true" effect estimates in different CoE levels.
Methods: We reasoned that stable treatment effect estimates over time indicate truthfulness. We compared odds ratios (ORs) from meta-analyses (MAs) before and after updates, expecting higher CoE to show similar ORs yielding a ratio of odds ratios (ROR) equal to 1. ROR=1 was predicted to be more common in higher CoE MAs. We analyzed 150 Cochrane MA pairs, using previously detailed methods (doi: 10.1111/jep.13657) to see if ROR = 1 (+/- a deviation margin) aligned with CoE levels according to GRADE methods. These MAs more likely represent “true” effects.
Results: Figure 1 shows a linear relationship between the probability of a true estimate of treatment effects as a function of CoE (at 10% margin error in ROR estimates) (R2=0.99; p=0.042) The probability of potentially "true" estimates decreases by about 20% (95%CI: 15 to 26%) for each drop in the rating of CoE. A linear relationship was less clear with a 5% ROR error margin, likely due to smaller sample size. Still, higher CoE showed significantly greater likelihood of true effects (53%) compared to non-high (i.e., moderate, low, or very low) CoE (25%); p=0.021.
Conclusion: This study offers the first estimate of potentially true treatment effects based on CoE. These findings can help refine decision-making in policy and clinical settings.