Article type
Year
Abstract
Background: Recently, Ioannidis1 and Wacholder et al2 estimated that most published research findings are false, but they did not indicate when, if at all, potentially false research results may become acceptable. We combined our two previously published models3,4 to calculate the probability above which research findings may become acceptable.
Methods: A new model indicates that a decision whether to accept the results of research findings (i.e. the alternative hypothesis) depends on the strength of evidence, a ratio of benefits/harms (B/H) related to intervention(s) tested, and our tolerance towards the penalty (regret) for being wrong in accepting (false) research results.
Results: The probability above which research results should be accepted depends on the research's expected payback (benefits) and inadvertent consequences (harms). For example, results from adequately powered randomized trials (RCTs) with little bias should be accepted even if the benefit/harm ratio is as small as 0.18; on the other hand, the B/H ratio should exceed 999 if we are rationally to accept the findings from discovery-oriented exploratory molecular profiling research. When we incorporated our willingness to tolerate the error in wrongly accepting the findings of a research study (regret) in the model, the results are dramatically different: when there is unwillingness to tolerate a loss of 1% of benefits in case we wrongly accepted research findings from RCTs, the minimum B/H ratio to accept such results increase to 15. However, if we are prepared to lose a greater fraction of potential benefits for being wrong, then the results of a research study may become tolerable under exceedingly small benefit/harm ratios, particularly if our prior beliefs in the correctness of the research hypothesis were very small (and hence the penalty for being wrong will remain small). If we are not willing to accept any possibility of being wrong, then research becomes acceptable only if we are absolutely certain in the 'truth' of the research hypothesis. This is clearly an unattainable goal. The table summarizes the results.
Conclusions: Since obtaining absolute 'truth' in research is impossible, society has to decide when less than perfect results may become acceptable. We have shown that this depends on benefit and harms associated with the research findings as well as our tolerance towards the error of wrongly accepting the results of a research hypothesis.
References
1. Ioannidis JP. Why most published research findings are false. PloS Medicine 2005;2(8):e124.
2. Wacholder S, Chanock S, Garcia-Closas M, El Ghormli L, Rothman N. Assessing the probability that a positive report is false: an approach for molecular epidemiology studies. Journal of the National Cancer Institute 2004;96(6):434-42.
3. Djulbegovic B, Hozo I. At what degree of belief in a research hypothesis is a trial in humans justified? Journal of Evaluation in Clinical Practice 2002;8(2):269-76.
4. Djulbegovic B, Hozo I, Schwartz A, McMasters KM. Acceptable regret in medical decision making. Medical Hypotheses 1999;
53(3):253-9.
Methods: A new model indicates that a decision whether to accept the results of research findings (i.e. the alternative hypothesis) depends on the strength of evidence, a ratio of benefits/harms (B/H) related to intervention(s) tested, and our tolerance towards the penalty (regret) for being wrong in accepting (false) research results.
Results: The probability above which research results should be accepted depends on the research's expected payback (benefits) and inadvertent consequences (harms). For example, results from adequately powered randomized trials (RCTs) with little bias should be accepted even if the benefit/harm ratio is as small as 0.18; on the other hand, the B/H ratio should exceed 999 if we are rationally to accept the findings from discovery-oriented exploratory molecular profiling research. When we incorporated our willingness to tolerate the error in wrongly accepting the findings of a research study (regret) in the model, the results are dramatically different: when there is unwillingness to tolerate a loss of 1% of benefits in case we wrongly accepted research findings from RCTs, the minimum B/H ratio to accept such results increase to 15. However, if we are prepared to lose a greater fraction of potential benefits for being wrong, then the results of a research study may become tolerable under exceedingly small benefit/harm ratios, particularly if our prior beliefs in the correctness of the research hypothesis were very small (and hence the penalty for being wrong will remain small). If we are not willing to accept any possibility of being wrong, then research becomes acceptable only if we are absolutely certain in the 'truth' of the research hypothesis. This is clearly an unattainable goal. The table summarizes the results.
Conclusions: Since obtaining absolute 'truth' in research is impossible, society has to decide when less than perfect results may become acceptable. We have shown that this depends on benefit and harms associated with the research findings as well as our tolerance towards the error of wrongly accepting the results of a research hypothesis.
References
1. Ioannidis JP. Why most published research findings are false. PloS Medicine 2005;2(8):e124.
2. Wacholder S, Chanock S, Garcia-Closas M, El Ghormli L, Rothman N. Assessing the probability that a positive report is false: an approach for molecular epidemiology studies. Journal of the National Cancer Institute 2004;96(6):434-42.
3. Djulbegovic B, Hozo I. At what degree of belief in a research hypothesis is a trial in humans justified? Journal of Evaluation in Clinical Practice 2002;8(2):269-76.
4. Djulbegovic B, Hozo I, Schwartz A, McMasters KM. Acceptable regret in medical decision making. Medical Hypotheses 1999;
53(3):253-9.