Background: According to the Cochrane Handbook, interpretation of heterogeneity estimates (I2 statistic) can be interpreted as: 1) ‘not important’ (0-40%); 2) ‘moderate heterogeneity’ (30-60%); 3) ‘substantial heterogeneity’ (50-90%); and 4) ‘considerable heterogeneity’ (75%-100%). Further, I2 should be interpreted according to strength of evidence for heterogeneity (i.e. p-value from the chi-squared test). There is a risk that meta-analyses with a limited number of trials may yield heterogeneity estimates (I2) that are subject to random error. Objectives: To assess how the I2 statistic and the chi-squared test for heterogeneity fluctuate according to the number of pooled trials. Methods: We searched The Cochrane Library, Issue 4, 2008 for meta-analyses that were among the first three outcomes in the first comparison group, reported a binary outcome, and pooled at least 30 trials. We only included one meta-analysis per systematic review. We calculated the number of meta-analyses where the minimum (maximum) non-significant I2 estimate and maximum (minimum) significant I2 estimates was at least one category apart. For example, in Figure 1 the minimum non-significant I2 estimate is 0%, and the maximum significant I2 estimate is 61%. We also calculated the number of meta-analyses where non-significant heterogeneity estimates fluctuated between at least three categories. Results: Seventeen of 26 eligible meta-analyses were significantly heterogeneous. In 10 meta-analyses, the minimum or maximum non-significant I2 estimates were at least one category apart from their significant counterpart. In five meta-analyses, non-significant heterogeneity estimates fluctuated between at least three categories. Significant heterogeneity estimates did not incur notable fluctuations. Conclusion: Heterogeneity estimates frequently incur considerable fluctuations before the test for heterogeneity becomes significant. Our findings support the recommendations of the Cochrane Handbook and underscore the importance of interpreting I2 in relation to the chi-squared test for heterogeneity.