Article type
Year
Abstract
Background: Meta-analysis of trials that have used different continuous or rating scales to record outcomes of a similar nature can be performed by pooling the trials' standardised mean difference (SMD). This is the difference in means divided by the pooled standard deviation.
Objectives: To study whether trial SMDs in meta-analyses are accurate.
Methods: Systematic review of meta-analyses published in 2004 that reported a result as an SMD. Two trials were randomly selected from each meta-analysis, and two observers independently extracted data. The main outcome measure was the proportion of meta-analyses where our calculated SMD differed from that of the authors by 0.1 or more, either for the point estimate, or for its confidence interval, for at least one of the two trials we selected from each meta-analysis. We chose 0.1 as cut-point as this may be an important error, considering the effect many commonly used treatments have compared with placebo. We contacted the authors when we could not replicate their results.
Results: We found 27 eligible meta-analyses on PubMed of which 16 were Cochrane reviews. We could not replicate the result within our cut-point for at least one of the two trials in 10 of the meta-analyses (37%). Four meta-analyses had a discrepancy of 0.60 or more for the point estimate. Common problems were erroneous number of patients, means, standard deviations and sign for the effect estimate. In total, 17 meta-analyses (63%) had data extraction errors although we checked only two trials per meta-analysis. For the 10 meta-analyses with important errors, we checked the data from all the trials in the meta-analysis result and did our own meta-analysis, using the authors' methods. For 7 of these meta-analyses, we could not replicate the authors' pooled result.
Conclusions: Meta-analyses based on SMDs should be read cautiously. Editors should consider checking the manuscripts for possible data extraction errors. Policy makers might wish to check the most pivotal data before making major policy decisions.
Objectives: To study whether trial SMDs in meta-analyses are accurate.
Methods: Systematic review of meta-analyses published in 2004 that reported a result as an SMD. Two trials were randomly selected from each meta-analysis, and two observers independently extracted data. The main outcome measure was the proportion of meta-analyses where our calculated SMD differed from that of the authors by 0.1 or more, either for the point estimate, or for its confidence interval, for at least one of the two trials we selected from each meta-analysis. We chose 0.1 as cut-point as this may be an important error, considering the effect many commonly used treatments have compared with placebo. We contacted the authors when we could not replicate their results.
Results: We found 27 eligible meta-analyses on PubMed of which 16 were Cochrane reviews. We could not replicate the result within our cut-point for at least one of the two trials in 10 of the meta-analyses (37%). Four meta-analyses had a discrepancy of 0.60 or more for the point estimate. Common problems were erroneous number of patients, means, standard deviations and sign for the effect estimate. In total, 17 meta-analyses (63%) had data extraction errors although we checked only two trials per meta-analysis. For the 10 meta-analyses with important errors, we checked the data from all the trials in the meta-analysis result and did our own meta-analysis, using the authors' methods. For 7 of these meta-analyses, we could not replicate the authors' pooled result.
Conclusions: Meta-analyses based on SMDs should be read cautiously. Editors should consider checking the manuscripts for possible data extraction errors. Policy makers might wish to check the most pivotal data before making major policy decisions.