Multiplicity of data in trial reports creates an important challenge for the reliability of meta-analyses: an empirical study

Article type
Authors
Tendal B1, Nüesch E2, Higgins J3, Jüni P2, Gøtzsche P1
1The Nordic Cochrane Centre, Copenhagen, Denmark
2University of Bern, Institute of Social and Preventive Medicine, Bern, Switzerland
3MRC Biostatistics Unit, University of Cambridge, Cambridge, UK
Abstract
Background: Authors performing meta-analyses of clinical trials often face a multiplicity of data in the trial reports. There may be several possible follow-up times, and the same outcome can be measured on different, but similar scales. The challenge of data multiplicity has not yet been examined in relation to meta-analyses. Objectives: We examined the scope for multiplicity in a sample of meta-analyses using the standardised mean difference (SMD) as an effect measure, and we examined the impact on the results. Methods: We selected all Cochrane reviews published in The Cochrane Library, issues 3, 2006 to 2, 2007, which presented a SMD. The firstSMDresult in each reviewwas used to identify a specific outcome for each meta-analysis in its protocol. Based on the protocols, two observers independently extracted data from the trial reports for any groups, outcome measures or time points compatible with the protocol. Based on these data, all possible SMDs were calculated in Monte Carlo simulations. Results: Eighty-three trials (19 meta-analyses) were included. Twenty-four (29%) trials reported data on multiple intervention groups, 30 (36%) provided data on multiple time points and 28 (34%) trials reported the index outcome measured on multiple scales. In 18 out of 19 meta-analyses, we found multiplicity of data in trial reports in at least one trial. Pooled SMD results were affected in 17 of 19 (89%) meta-analyses. The median variability across meta-analyses was a median difference between two randomly selected SMDs within the same meta-analysis of 0.11 standard deviation units (range 0.03 to 0.41). Conclusions: Multiplicity can impact importantly on meta-analyses. To reduce the risk of bias in reviews, protocols should pre-specify which results are preferred in relation to time points, intervention groups and scales.