Article type
Year
Abstract
Background: Divergent results on the same question generate controversy. The relative timing of the appearance of such discrepancies in the published literature has not been assessed. This may be evaluated in the context of meta-analyses.
Objectives: To evaluate the hypothesis that the most extreme, opposite results are likely to appear very early rather than late as data accumulate, provided that a lot of data can be generated rapidly.
Methods: We evaluated the potential presence of this bias in a set of 44 cumulative meta-analyses (534 studies) on associations of genetic risk factors with complex diseases. These represent a hot scientific field where data can be generated very rapidly. As a control we used a database of 37 sampled cumulative meta-analyses of randomized controlled trials (RCTs, n=384) - clinical research is typically more unwieldy and time-consuming. In cumulative meta-analysis, summary estimates of effect size as well as the extent of heterogeneity between results of the constituent studies are updated with new data at the end of each year. In these datasets we recorded the relative timing of publication of the most extreme opposite results and serial estimates of between-study heterogeneity over time, as more evidence was published.
Results: In genetic meta-analyses it was more likely for the most favorable-ever result than the least favorable-ever result to appear during the first heterogeneity assessment (in 17 vs. 5 cumulative meta-analyses, P=0.017; in 5 cases both extremes appeared upfront). At the time of the second heterogeneity assessment, it was significantly more likely for the least favorable-ever result than for the most favorable-ever result to appear (4 vs. 14, P=0.031; in 1 case both extremes appeared then). On the contrary, no such behavior was seen in meta-analyses of RCTs (P=0.21 and P=0.77, respectively).
At least some between-study heterogeneity was recorded in 40/44 vs. 32/37 meta-analyses at least once during the accumulation of evidence. The maximal between-study variance was significantly more likely to be recorded early in genetic associations than in health care interventions (31/40 vs. 15/32, respectively, occurred within the first two heterogeneity assessments, P=0.013).
Conclusions: Extreme opposite results may be easier to generate in molecular medicine than in controlled clinical research. Extreme opposing views may appear in rapid succession in research, when there is large genuine heterogeneity or rapid data production.
Acknowledgements: Supported in part by a PENED grant from the General Secretariat for Research and Technology, Greece and the European Commission.
Objectives: To evaluate the hypothesis that the most extreme, opposite results are likely to appear very early rather than late as data accumulate, provided that a lot of data can be generated rapidly.
Methods: We evaluated the potential presence of this bias in a set of 44 cumulative meta-analyses (534 studies) on associations of genetic risk factors with complex diseases. These represent a hot scientific field where data can be generated very rapidly. As a control we used a database of 37 sampled cumulative meta-analyses of randomized controlled trials (RCTs, n=384) - clinical research is typically more unwieldy and time-consuming. In cumulative meta-analysis, summary estimates of effect size as well as the extent of heterogeneity between results of the constituent studies are updated with new data at the end of each year. In these datasets we recorded the relative timing of publication of the most extreme opposite results and serial estimates of between-study heterogeneity over time, as more evidence was published.
Results: In genetic meta-analyses it was more likely for the most favorable-ever result than the least favorable-ever result to appear during the first heterogeneity assessment (in 17 vs. 5 cumulative meta-analyses, P=0.017; in 5 cases both extremes appeared upfront). At the time of the second heterogeneity assessment, it was significantly more likely for the least favorable-ever result than for the most favorable-ever result to appear (4 vs. 14, P=0.031; in 1 case both extremes appeared then). On the contrary, no such behavior was seen in meta-analyses of RCTs (P=0.21 and P=0.77, respectively).
At least some between-study heterogeneity was recorded in 40/44 vs. 32/37 meta-analyses at least once during the accumulation of evidence. The maximal between-study variance was significantly more likely to be recorded early in genetic associations than in health care interventions (31/40 vs. 15/32, respectively, occurred within the first two heterogeneity assessments, P=0.013).
Conclusions: Extreme opposite results may be easier to generate in molecular medicine than in controlled clinical research. Extreme opposing views may appear in rapid succession in research, when there is large genuine heterogeneity or rapid data production.
Acknowledgements: Supported in part by a PENED grant from the General Secretariat for Research and Technology, Greece and the European Commission.