Exploring discrepancies in the findings of 20 systematic reviews addressing the same PICO question: a case study

Khamis AM1, El Moheb M2, Nicolas J2, Refaat M3, Akl EA3
1Clinical Research Institute, American University of Beirut, 2Faculty of Medicine, American University of Beirut, 3Department of Internal Medicine, American University of Beirut

Background:

There are sometimes variation(s) in the meta-analytical findings between systematic reviews that address the same question and include the same studies.

Objective:

To assess and investigate the reasons for discrepancies between the results of meta-analyses from systematic reviews addressing the same question.

Methods:

We conducted a Cochrane Review on 'Implantable cardiac defibrillators for patients with non-ischemic cardiomyopathy' with all-cause mortality as the primary outcome. In parallel, we searched for non-Cochrane systematic reviews addressing the same question. We abstracted detailed information about number of studies, data extracted, analytical methods and effect estimates. We assessed the variation between the pooled estimates in meta-analyses through standard deviation calculation and we investigated the reasons for the discrepancies.

Results:

We identified 20 non-Cochrane systematic reviews published in 2017 that addressed the same question. All 20 reviews were published in 2017 and ran their searches after the publication of all included trials. The standard deviation for the variation of the pooled effect estimate for all cause-mortality across the 20 non-Cochrane systematic reviews was 4%.

We identified the following as reasons for the discrepancies in the findings of the 20 systematic reviews: 1) different included trials, 2) different effect estimates used (i.e. odds ratio (OR), relative risk (RR) or hazard ratio (HR)) in meta-analysis, 3) different effect estimates used from trials, 4) different analysis models used (random- or fixed-effect), 5) errors in data abstraction or entry, 6) errors in reporting of results, 7) obtaining additional data directly from trial authors and 8) use of different outcome timeframes. For example, only seven systematic reviews (35%) included all six trials in their meta-analyses. Half of the systematic reviews pooled RRs, while 40% and 15% pooled HRs and ORs, respectively.

Conclusion:

In a sample of 21 systematic reviews addressing the same question, there were frequent discrepancies between their meta-analytical results for the same outcome. It is debatable whether these discrepancies have implications for the interpretation of the findings. Reasons for the discrepancies were related either to errors or to choices in the analytical approach.

Patient or healthcare consumer involvement:

No involvement.