Cochrane versus non-Cochrane reviews: an exploration of discordance, comparative quality and relevance for patient information

Tags: Oral
Bastian H, Knelangen M, Lange S, Nasser M

Background: The German Institute for Quality and Efficiency in Health Care (IQWiG) summarises over 100 systematic reviews per year for patients on the website Informed Health Online/Gesundheitsinformation.de. For reviews initially selected, a search for other reviews in the last three or five years is conducted. If there are multiple reviews, a comparison of quality is made. The review determined to be of the best quality and most relevant for patients is chosen. Objectives: To explore the reasons for discordance among multiple systematic reviews on the same subject, and to identify how often and why Cochrane reviews (CRs) were selected over non-Cochrane reviews. Methods: A consecutive series of 50 intervention topics selected in 2007 for inclusion on the website was analysed. The following data were extracted: total number of ‘competing’ reviews considered; whether it was a CR; reason for selection or rejection of CRs; and reasons logged for discordance identified among the reviews. Results: For 50 topics, a total of 135 reviews on the same subjects were considered. There were nine instances of important discordance in conclusions among 32 cases with multiple reviews. (These nine reviews were among the 15 cases where there were more than three reviews). Some related to the inclusion of different trials or consideration of different endpoints or populations, and some related to differing interpretations of similar data. Conflicts of interest may have played a role on two occasions. Of the 50 finally chosen, 24 were based on CRs (48%). In 16 of the 26 cases where non-CRs were chosen, there was no CR on that topic. The most recent review was not always chosen as later trials were not always likely to have affected review conclusions. However, in the 10 cases where non-CRs were selected over CRs, on five occasions this was primarily because the CR was outdated. Reviews with broader topic coverage were preferred over narrow review questions. Conclusions: Different inclusion criteria did not always explain discordance between systematic review results. Discordance was more common when there were more than three reviews on the same subject. Implications for Cochrane reviewers will be discussed.