Article type
Year
Abstract
Background: For a particular outcome in a systematic review, there may be a multiplicity of data available in the trial reports (e.g. multiple scales, time points, and analyses). Multiplicity can potentially lead to data driven decisions regarding which data to include. Pre-specified decision rules known as ‘outcome data hierarchies’ (e.g. listing eligible scales in order of preferred inclusion) are a suggested method to deal with multiplicity.
Objective: To investigate how often systematic reviewers pre-specify outcome data hierarchies and the impact of such hierarchies on the extent of data available for inclusion in a meta-analysis.
Methods: Cochrane and non-Cochrane systematic reviews published from January 2010 to January 2012 were randomly sampled. All trial data that were compatible with (i) the review definition of the first meta-analysed continuous outcome, and (ii) the hierarchies reported in the review protocol, where available, were extracted. The proportion of (i) trials with multiplicity, (ii) reviewers pre-specifying hierarchies, and (iii) reviews including at least one trial with multiplicity, were quantified.
Results: Forty-four reviews including 276 trials were evaluated. Multiplicity was present in half the trials 95% confidence interval (CI) 44–57%; multiple scales and intervention groups were most common. In reviews with a published protocol (N = 21), hierarchies were pre-specified in relation to the following: intervention groups (0%); scales (19%); time points (62%); intention-to-treat versus alternative analyses (71%); and final versus change from baseline values (29%). Reviews with at least one pre-specified hierarchy had a lower chance of including a trial with multiplicity compared to reviews with no pre-specified hierarchies (9/17 vs. 23/27; risk difference −0.32, 95%CI −0.59 to −0.05).
Conclusion: Use of outcome data hierarchies varies across systematic reviews. Pre-specifying hierarchies can reduce the number of trial effect estimates available for inclusion in a meta-analysis, and prevent selective inclusion based on the results.
Objective: To investigate how often systematic reviewers pre-specify outcome data hierarchies and the impact of such hierarchies on the extent of data available for inclusion in a meta-analysis.
Methods: Cochrane and non-Cochrane systematic reviews published from January 2010 to January 2012 were randomly sampled. All trial data that were compatible with (i) the review definition of the first meta-analysed continuous outcome, and (ii) the hierarchies reported in the review protocol, where available, were extracted. The proportion of (i) trials with multiplicity, (ii) reviewers pre-specifying hierarchies, and (iii) reviews including at least one trial with multiplicity, were quantified.
Results: Forty-four reviews including 276 trials were evaluated. Multiplicity was present in half the trials 95% confidence interval (CI) 44–57%; multiple scales and intervention groups were most common. In reviews with a published protocol (N = 21), hierarchies were pre-specified in relation to the following: intervention groups (0%); scales (19%); time points (62%); intention-to-treat versus alternative analyses (71%); and final versus change from baseline values (29%). Reviews with at least one pre-specified hierarchy had a lower chance of including a trial with multiplicity compared to reviews with no pre-specified hierarchies (9/17 vs. 23/27; risk difference −0.32, 95%CI −0.59 to −0.05).
Conclusion: Use of outcome data hierarchies varies across systematic reviews. Pre-specifying hierarchies can reduce the number of trial effect estimates available for inclusion in a meta-analysis, and prevent selective inclusion based on the results.