Article type
Year
Abstract
Background:
For a particular meta-analysis, there may be multiple effect estimates available to select from a trial report, resulting, for example, from multiple scales, time points, or analyses. Multiplicity can potentially lead to data driven decisions regarding which data to include in a meta-analysis, which we refer to as “selective inclusion of results”. Selective inclusion of results may bias meta-analytic effect estimates.
Objective:
To investigate whether there is evidence of selective inclusion of results in systematic reviews, and what impact it has on meta-analytic effects.
Methods:
Systematic reviews published between January 2010-2012 were randomly sampled. Trial effect estimates that were available for inclusion in the first continuous outcome meta-analysis in each review were extracted. For each trial, extracted effect estimates were ranked according to magnitude, and the selected effect estimate’s location between the lowest and highest rank was calculated. A Potential Bias Index (PBI) was developed to investigate whether there was evidence of selective inclusion. The PBI quantifies the average ranking location of the trial effect estimates selected for inclusion in a meta-analysis (where PBI=1 when the highest ranked effect estimate in each trial is always selected, PBI=0 when the lowest ranked effect estimate is always selected, and PBI=0.5 under a process consistent with random selection).
Results:
Thirty-one reviews including 228 trials were evaluated. Half the trials had multiple effect estimates that were available for inclusion in a particular meta-analysis. The PBI was 0.58 (95% confidence interval 0.51 to 0.65; two-tailed p-value of 0.049), providing some evidence that on average, larger effect estimates were included than we would expect through a process consistent with random selection. Analysis of the potential impact of selective inclusion of results on meta-analytic effect estimates is ongoing and will be presented at the Colloquium.
Conclusion:
This is the first study to explore selective inclusion of results in systematic reviews. The findings may inform guidance for dealing with multiplicity of data in trial reports.
For a particular meta-analysis, there may be multiple effect estimates available to select from a trial report, resulting, for example, from multiple scales, time points, or analyses. Multiplicity can potentially lead to data driven decisions regarding which data to include in a meta-analysis, which we refer to as “selective inclusion of results”. Selective inclusion of results may bias meta-analytic effect estimates.
Objective:
To investigate whether there is evidence of selective inclusion of results in systematic reviews, and what impact it has on meta-analytic effects.
Methods:
Systematic reviews published between January 2010-2012 were randomly sampled. Trial effect estimates that were available for inclusion in the first continuous outcome meta-analysis in each review were extracted. For each trial, extracted effect estimates were ranked according to magnitude, and the selected effect estimate’s location between the lowest and highest rank was calculated. A Potential Bias Index (PBI) was developed to investigate whether there was evidence of selective inclusion. The PBI quantifies the average ranking location of the trial effect estimates selected for inclusion in a meta-analysis (where PBI=1 when the highest ranked effect estimate in each trial is always selected, PBI=0 when the lowest ranked effect estimate is always selected, and PBI=0.5 under a process consistent with random selection).
Results:
Thirty-one reviews including 228 trials were evaluated. Half the trials had multiple effect estimates that were available for inclusion in a particular meta-analysis. The PBI was 0.58 (95% confidence interval 0.51 to 0.65; two-tailed p-value of 0.049), providing some evidence that on average, larger effect estimates were included than we would expect through a process consistent with random selection. Analysis of the potential impact of selective inclusion of results on meta-analytic effect estimates is ongoing and will be presented at the Colloquium.
Conclusion:
This is the first study to explore selective inclusion of results in systematic reviews. The findings may inform guidance for dealing with multiplicity of data in trial reports.