A case study of diminishing returns: updating a hypothesis-generating systematic review with 'negative' findings

Article type
Paul Dellavalle R, Mary Schilling L, R. Freeman S
Background: More than 5 years ago, publications of two randomized placebo-controlled trials respectively assessing lovastatin (a statin) and gemfibrozil (a fibrate) for cardiovascular disease prevention, reported significantly lower incidence of melanoma, the deadliest form of skin cancer, in participants receiving these lipid-lowering drugs. Due to increased liver cancer in rodents fed statins and fibrates, trials in humans have commonly monitored cancer incidence. A systematic review of all similar trials (prior to 2004) showed no association of statin or fibrate exposure with lower melanoma incidence (Cochrane Database Syst Rev. 2005 Issue 4).
Objectives: To explore methodological and logistic concerns regarding updating a hypothesis-generating systematic review with 'negative' findings.
Methods: Trial inclusion criteria for the update were unchanged. Qualifying trials a) randomized participants, b) possessed at least one trial arm that used the lipid-lowering medication without other medications, and c) had a mean participant therapeutic duration of at least 4 years. New studies were identified by repeating original search strategies for articles published between 2003 and 2006, and by examining qualifying trial references. Melanoma data were extracted from publications, and if unpublished, requested from the corresponding authors. Data were also requested from the corresponding authors of three original qualifying trials that did not provide data for the original systematic review. Monetary incentive for providing data to the initial review of ($50 US per case of melanoma) was not offered for the update.
Results: The original review identified 16 qualifying trials (7 statin, 9 fibrate) of which 13 provided data on incident melanomas (6 statin, 7 fibrate). Two trials requested and received monetary incentive payments for their data. The three trials not providing data to the original review did not provide data to the update. The new search produced 4,403 article titles that were screened for relevance by two reviewers yielding 234 abstracts culled to 44 full articles for review. Ultimately 9 new qualifying trials were identified (7 statin, 2 fibrate) of which 2 statin trials provided new melanoma data-17 cases in participants taking a statin and 10 in those taking a placebo. Two trialists declined to provide new data due to the negative results of the initial systematic review. Overall a total of 153 melanomas occurred among 51,937 participants in the 8 statin trials (76 among the 26,056 statin group participants and 77 among the 25,881 control group participants). No new fibrate data was obtained by the update--as previously reported, a total of 26 melanomas occurred among the 30,973 participants in the fibrate trials (7 among the 12,120 fibrate group participants and 19 among the 18853 control group participants). Incidence of melanoma was not associated with the use of statins (OR = 1.02, 95% CI = 0.65 to 1.61) or fibrates (OR = 0.58, 95% CI = 0.19 to 1.82).
Conclusions: Updated results are limited due to the inability to include all relevant trial data as melanoma data were obtained from a lower proportion of trials identified for the update (2/9) than for the original review (13/16). Updated results continue to not support the hypothesis that standard doses of statins or fibrates are associated with lower melanoma incidence. Our experiences support further research on the following questions: How often should systematic reviews of unorthodox hypotheses with 'negative results' be updated? What is the effect of a) monetary incentives and b) published 'negative' results on data collection? How can overworked trialists be further coaxed to provide unpublished data? And should the Cochrane Collaboration support a de-identified Internet archive for clinical trial information that would foster systematic review data collection on unanticipated secondary outcomes?