Article type
Year
Abstract
Background: Understanding how researchers/clinicians interpret evidence may improve communication of meta-analysis results.
Objective: To explore discrepancies in treatment recommendations based on identical meta-analyses.
Methods: For the quantitative component of our mixed-methods study, we simulated meta-analysis data after 3, 5, 10 and 20 studies for 4 different scenarios. Based on these data (included individual study results, fixed and random effects point estimates, I2, published meta-analysts estimated the treatment effect at each point, and whether the treatment was shown beneficial, or would be shown beneficial in the future, and if they would recommend treatment. For the qualitative component, we conducted individual semi-structured interviews with the meta-analysts to substantiate their quantitative responses.
Results: These results are based on the 14 meta-analysts analyzed thus far. In general, participants did not believe a treatment was beneficial if there were ≤5 studies. An exploratory Classification and Regression Tree analysis suggested recommending treatment was mostly dependent on the meta-analyst believing the treatment would be proven beneficial in the future, which itself was determined by the treatment being shown beneficial at that time. The total number of events was the most important factor determining if treatment was considered beneficial, provided the confidence interval did not cross the null (86% recommended and 14% unsure). When the confidence interval included the null, 63% of meta-analysts did not recommend treatment. When total events were low, those with higher risk-taking behaviour were more likely to consider (Yes or Unsure) recommending treatment (71% vs. 16%). Our qualitative analysis corroborated the quantitative findings.
Conclusions: Decisions to recommend treatment based on meta-analyses appear to be generally related to the number of total events in the meta-analysis, and whether the confidence interval crosses the null.
Objective: To explore discrepancies in treatment recommendations based on identical meta-analyses.
Methods: For the quantitative component of our mixed-methods study, we simulated meta-analysis data after 3, 5, 10 and 20 studies for 4 different scenarios. Based on these data (included individual study results, fixed and random effects point estimates, I2, published meta-analysts estimated the treatment effect at each point, and whether the treatment was shown beneficial, or would be shown beneficial in the future, and if they would recommend treatment. For the qualitative component, we conducted individual semi-structured interviews with the meta-analysts to substantiate their quantitative responses.
Results: These results are based on the 14 meta-analysts analyzed thus far. In general, participants did not believe a treatment was beneficial if there were ≤5 studies. An exploratory Classification and Regression Tree analysis suggested recommending treatment was mostly dependent on the meta-analyst believing the treatment would be proven beneficial in the future, which itself was determined by the treatment being shown beneficial at that time. The total number of events was the most important factor determining if treatment was considered beneficial, provided the confidence interval did not cross the null (86% recommended and 14% unsure). When the confidence interval included the null, 63% of meta-analysts did not recommend treatment. When total events were low, those with higher risk-taking behaviour were more likely to consider (Yes or Unsure) recommending treatment (71% vs. 16%). Our qualitative analysis corroborated the quantitative findings.
Conclusions: Decisions to recommend treatment based on meta-analyses appear to be generally related to the number of total events in the meta-analysis, and whether the confidence interval crosses the null.