Multiplicity of data in trial reports: an important challenge for the reliability of meta-analyses

Article type
Authors
Tendal B, Nüesch E, Higgins J, Jüni P, C Gøtzsche P
Abstract
Background: Authors performing meta-analyses will often face a multiplicity of data in the trial reports. There may be several possible follow-up times, and the same outcome can be measured on different, but similar scales. The challenge of data multiplicity has not yet been examined. Objectives: To examine the scope for multiplicity in a sample of meta-analyses using the standardised mean difference (SMD) method and to examine the impact of the multiplicity on the results. Methods: We selected all Cochrane reviews published in The Cochrane Library in Issues 3, 2006 to 2, 2007 that presented a result as an SMD. We retrieved the trial reports that corresponded to the first SMD result in each review and retrieved the review protocols. These index SMDs were used to identify the relevant outcome for each meta-analysis. Based on the protocols and this outcome, two observers independently extracted the necessary data from the trial reports for calculation of the SMDs. Any information on which control groups to select was also used. Based on the extracted data, all possible SMDs were calculated in Monte Carlo simulations, allowing only one source of multiplicity to vary at a time, to estimate the impact of each source of multiplicity. Results: Preliminary results based on 19 meta-analyses including 84 trials, show that review protocols in many instances lack information about which data to choose. Sixteen of the meta-analyses were affected by multiplicity and the size of the pooled SMDs varied substantially. The sources of multiplicity were: time points (n=13), groups (n=9) and scales (n=8) (11 meta-analyses had more than one source). Conclusions: Multiplicity can impact importantly on meta-analyses. To reduce the risk of bias in reviews, it should be pre-specified in protocols and reported in publications which results are preferred in relation to time points, groups and scales.