Article type
Year
Abstract
Background:
It is still widely believed that the P value is the probability that your results occurred by chance. This is simply wrong. It confuses the P value with the false positive risk (FPR).
Objectives:
The aim is to reduce the number of false positive results that are published.
Methods and Results:
If you observe a ‘significant’ P value after doing a single unbiased experiment, what is the probability that your result is a false positive?
In order to calculate the false positive risk, you need Bayes’ theorem. That involves the prior probability that your hypothesis is true, i.e. the probability that there is a real effect there before the experiment is done. You hardly ever have a value for this, so what can be done? There are two possibilities.
First, you can say that a prior probability bigger than 0.5 is hardly ever justified, so you can calculate a minimum FPR based on the assumption of a prior of 0.5. If you observe P = 0.05 then the FPR is at least 26%. If the hypothesis was implausible, with a prior of 0.1, then then the FPR would be a disastrous 76%.
Second, since you don’t know the prior, you can calculate it as the value that would be needed to reduce the FPR to 0.05 (that is what most people still think that the P value tells you). If you observe P = 0.05 in a well-powered experiment then you would need to assume a prior of 0.87 in order to achieve an FPR of 0.05. You would have to assume that you were almost (87%) certain that there was a real effect before you did the experiment.
Conclusions:
It's suggested that P values and confidence intervals should still be given, but they should be supplemented by a single extra number that conveys the strength of evidence better than the P value. This could be either the prior needed for an FPR of 0.05, or the minimum FPR.
The assumptions that underlie these calculations are described in:
http://rsos.royalsocietypublishing.org/content/4/12/171085
The calculations are done most easily with the web calculator at:
http://fpr-calc.ucl.ac.uk
It is still widely believed that the P value is the probability that your results occurred by chance. This is simply wrong. It confuses the P value with the false positive risk (FPR).
Objectives:
The aim is to reduce the number of false positive results that are published.
Methods and Results:
If you observe a ‘significant’ P value after doing a single unbiased experiment, what is the probability that your result is a false positive?
In order to calculate the false positive risk, you need Bayes’ theorem. That involves the prior probability that your hypothesis is true, i.e. the probability that there is a real effect there before the experiment is done. You hardly ever have a value for this, so what can be done? There are two possibilities.
First, you can say that a prior probability bigger than 0.5 is hardly ever justified, so you can calculate a minimum FPR based on the assumption of a prior of 0.5. If you observe P = 0.05 then the FPR is at least 26%. If the hypothesis was implausible, with a prior of 0.1, then then the FPR would be a disastrous 76%.
Second, since you don’t know the prior, you can calculate it as the value that would be needed to reduce the FPR to 0.05 (that is what most people still think that the P value tells you). If you observe P = 0.05 in a well-powered experiment then you would need to assume a prior of 0.87 in order to achieve an FPR of 0.05. You would have to assume that you were almost (87%) certain that there was a real effect before you did the experiment.
Conclusions:
It's suggested that P values and confidence intervals should still be given, but they should be supplemented by a single extra number that conveys the strength of evidence better than the P value. This could be either the prior needed for an FPR of 0.05, or the minimum FPR.
The assumptions that underlie these calculations are described in:
http://rsos.royalsocietypublishing.org/content/4/12/171085
The calculations are done most easily with the web calculator at:
http://fpr-calc.ucl.ac.uk