Article type
Abstract
Background
Until recently the Cochrane Handbook had suggested that the I² index be employed to quantify heterogeneity. However, recent changes to the handbook encourage researchers to use the prediction interval for this purpose. This is a very important development and can substantially alter the way we understand the clinical utility of interventions.
Objectives
The vast majority of meta-analyses employ the I² index to quantify heterogeneity. Often, researchers use these statistics to classify the heterogeneity as being low, moderate, or high and then use these classifications when considering the potential utility of the intervention. While this practice is ubiquitous, it is nevertheless incorrect. The I² index does not tell us how much the effect size varies, and these classifications are not informative.
By contrast, the prediction interval tells us how much the effect size varies in a way that (a) actually addresses the amount of dispersion, (b) reports it in the relevant metric, and (c) provides the information that we need to make informed decisions about the clinical utility or substantive impact of an intervention. It provides the information that many researchers believe (incorrectly) is provided by I².
I will use an example from COVID to demonstrate how prediction intervals can tell us about the clinical utility of a treatment in a way that the I² index cannot. In this analysis, using I² we would conclude that the treatment is associated with increased survival and that this is statistically significant. By contrast, using the prediction interval we conclude that the treatment is associated with increased survival in 65% of cases but with increased mortality in the other 35%. In a case like this it can be a serious mistake to focus on the fact that the treatment is associated with increased survival “on average.” Rather, we need to understand when the treatment may be helpful and when it may be harmful.
Conclusions
My goal in this presentation is to explain what I² and the prediction interval tell us. This will provide clarity for researchers, clinicians, and consumers as Cochrane begins to transition to the wider use of prediction intervals.
Until recently the Cochrane Handbook had suggested that the I² index be employed to quantify heterogeneity. However, recent changes to the handbook encourage researchers to use the prediction interval for this purpose. This is a very important development and can substantially alter the way we understand the clinical utility of interventions.
Objectives
The vast majority of meta-analyses employ the I² index to quantify heterogeneity. Often, researchers use these statistics to classify the heterogeneity as being low, moderate, or high and then use these classifications when considering the potential utility of the intervention. While this practice is ubiquitous, it is nevertheless incorrect. The I² index does not tell us how much the effect size varies, and these classifications are not informative.
By contrast, the prediction interval tells us how much the effect size varies in a way that (a) actually addresses the amount of dispersion, (b) reports it in the relevant metric, and (c) provides the information that we need to make informed decisions about the clinical utility or substantive impact of an intervention. It provides the information that many researchers believe (incorrectly) is provided by I².
I will use an example from COVID to demonstrate how prediction intervals can tell us about the clinical utility of a treatment in a way that the I² index cannot. In this analysis, using I² we would conclude that the treatment is associated with increased survival and that this is statistically significant. By contrast, using the prediction interval we conclude that the treatment is associated with increased survival in 65% of cases but with increased mortality in the other 35%. In a case like this it can be a serious mistake to focus on the fact that the treatment is associated with increased survival “on average.” Rather, we need to understand when the treatment may be helpful and when it may be harmful.
Conclusions
My goal in this presentation is to explain what I² and the prediction interval tell us. This will provide clarity for researchers, clinicians, and consumers as Cochrane begins to transition to the wider use of prediction intervals.