Definitely Maybe: How is uncertainty expressed in Cochrane reviews?

Tags: Oral
Smith A, Middleton P, Wager E

Abstract: Many Cochrane reviews produce clear-cut, unequivocal conclusions. However, many do not. It is vital that we do not abuse the power and authority of the Cochrane Library by claiming to have an answer where none exists. Indeed, reinforcing the message that clinical uncertainty is present may be beneficial [1]. Hence the issue of how we express and deal with uncertainty is crucial. We have surveyed a selection of reviews from Issue 2 (2000) to explore how uncertainty is addressed. We found that the form of words used to express uncertainty varies considerably. A few illustrative and anonymised examples are given here (our italics): * 'There is not enough evidence to evaluate the use of ...' / 'There is insufficient evidence to confirm..' These beg the question of how much evidence is enough - is there a minimum number of trials for a meaningful meta-analysis, for instance? * 'There is no evidence that unselected use of....'/ 'There are no data from randomised controlled trials to provide evidence for or against current practices' * Though the end result is the same i.e. both reviews are inconclusive, these two formulations may imply different things. Whilst the first could mean either that there really is no evidence to be found, or that what evidence there is does not help answer the review question, the second is more specific and states quite clearly the basis for the conclusion. * '[intervention X] appears to reduce the frequency of....'/ ' [these measures] seem to be ineffective...' * These can be confusing. Has this form of words been chosen because the evidence is equivocal, or because the reviewers are ambivalent about what they have found? Or do they prefer to 'sit on the fence' rather than make a bold, unambiguous conclusion which might later be disproven when further trials become available? * 'There is no clear evidence..'/ 'There is no firm evidence...'/ 'The best available evidence suggests....' Again, these words are subjective, value-laden and lack meaning without further qualification. Our presentation will develop and expand on these ideas and offer some suggestions for discussion. 1. Alderson P, Roberts I. Should journals publish systematic reviews that find no evidence to guide practice? Examples from injury research. BMJ 2000; 320: 376-7