Article type
Year
Abstract
Background:
P values and confidence intervals (CIs) provide different information. P values provide information about statistical significance while CIs add information on the direction and magnitude of the effect. Considered in isolation, P values can be misleading, and lead to health decisions based exclusively on statistics.
Objectives:
To assess the number of studies using the term 'statistically significant' but not mentioning any CI for reporting comparisons in abstracts in The Cochrane Database of Systematic Reviews (Cochrane DSR) and in the 5 major general medical journals during the last decade.
Methods:
We searched in PubMed (15 March 2014) by year and by each of the top five journals according to their 2013 impact factor (New England Journal of Medicine (NEJM), Lancet, Plos Medicine (PM), BMJ, JAMA) and Cochrane DSR, from the year 2004 to 2014 using the search string: ((statistically significant[tiab]) NOT (“Confidence Interval”[tiab] or “Interval, Confidence”[tiab] or “Intervals, Confidence”[tiab] or 95% CI[tiab])) AND [Jour].
Results:
There was not a clear temporal trend in the number of abstracts that used the term 'statistically significant' without corresponding CIs in the Cochrane DSR, although there was a negative trend in relative terms (Table 1, Figure 1). In the five major general medical journals the problem was less frequent in absolute numbers, but bigger in relative terms than the Cochrane DSR in NEJM, Lancet and PM (Figure 2).
Conclusions:
The potential misleading interpretation of reporting statistical significance without CIs is a problem that shows a slight downward trend over the last decade in the Cochrane DSR, but not a clear pattern among the top-five rated general medical journals. There have been clear recommendations about this problem in the Cochrane Handbook for Systematic Reviews of Interventions since September 2008, but statements are not so clear in the other journals. Probably clearer guidelines for authors would contribute to a solution.
P values and confidence intervals (CIs) provide different information. P values provide information about statistical significance while CIs add information on the direction and magnitude of the effect. Considered in isolation, P values can be misleading, and lead to health decisions based exclusively on statistics.
Objectives:
To assess the number of studies using the term 'statistically significant' but not mentioning any CI for reporting comparisons in abstracts in The Cochrane Database of Systematic Reviews (Cochrane DSR) and in the 5 major general medical journals during the last decade.
Methods:
We searched in PubMed (15 March 2014) by year and by each of the top five journals according to their 2013 impact factor (New England Journal of Medicine (NEJM), Lancet, Plos Medicine (PM), BMJ, JAMA) and Cochrane DSR, from the year 2004 to 2014 using the search string: ((statistically significant[tiab]) NOT (“Confidence Interval”[tiab] or “Interval, Confidence”[tiab] or “Intervals, Confidence”[tiab] or 95% CI[tiab])) AND [Jour].
Results:
There was not a clear temporal trend in the number of abstracts that used the term 'statistically significant' without corresponding CIs in the Cochrane DSR, although there was a negative trend in relative terms (Table 1, Figure 1). In the five major general medical journals the problem was less frequent in absolute numbers, but bigger in relative terms than the Cochrane DSR in NEJM, Lancet and PM (Figure 2).
Conclusions:
The potential misleading interpretation of reporting statistical significance without CIs is a problem that shows a slight downward trend over the last decade in the Cochrane DSR, but not a clear pattern among the top-five rated general medical journals. There have been clear recommendations about this problem in the Cochrane Handbook for Systematic Reviews of Interventions since September 2008, but statements are not so clear in the other journals. Probably clearer guidelines for authors would contribute to a solution.