Perpetuation of inappropriate meta-analysis methods? Analysis of systematic reviews cited by systematic reviews

Article type
Authors
Li T1, Swaroop-Vedula S1, Chang D1, Ervin A1, Wieland S2, Scherer R1, Dickersin K1, Montgomery P3
1Epidemiology, Johns Hopkins Bloomberg School of Public Health, Baltimore, Maryland, United States
2Complementary Medicine Field, Providence, Rhode Island, United States
3Barnett House, University of Oxford, Oxford, UK
Abstract
Objective: To describe citation patterns of multiple systematic reviews on a related topic. Methods: We included systematic reviews of prostaglandin analog eye drops for glaucoma identified by searching PubMed, EMBASE, and The Cochrane Library up to September 2009 without date or language restrictions. We extracted from each systematic review the number of studies and participants included, publication date, journal, funding sources, and whether a meta-analysis was conducted. We examined whether the statistical methods for meta-analysis were appropriate. One author extracted the data and another verified the data against the original publication. For each systematic review, we examined the reference list to identify all citations to previously published systematic reviews on the related topic. We also searched the Web of Science database on March 25, 2010 to verify citations and obtain the total number of citations for each review. Results: We identified 16 systematic reviews with meta-analysis (Figure). One or more inappropriate statistical methods were used in 8/16 reviews. Six described pooling data from similar treatment arms across studies resulting in a non-random comparison (e.g., six bimatoprost, and 11 travoprost arms were pooled and compared in a review including 27 trials); three described an incorrect formula to calculate the variance of the effect estimate. Systematic reviews using inappropriate meta-analysis methods were cited more often by subsequent reviews compared with systematic reviews using appropriate methods, many of which were not cited at all. The most frequently cited review (71 times) incorrectly pooled individual treatment arms. No review using inappropriate methods cited a review with appropriate methods. Reviews using appropriate methods did not cite reviews with inappropriate methods as often. Discussion: A critical assessment of previous systematic reviews on a given topic is imperative before initiating another. Additional systematic review topics should be examined for possible perpetuation of inappropriate statistical methods and findings.