Article type
Year
Abstract
Introduction: The use of meta-analysis in health-care decision-making is on the rise. The Cochrane Collaboration has taken a lead role in conducting and disseminating meta-analyses, and in the process has standardized the way these results are presented. However, there has been no empirical evidence collected to suggest that the Cochrane display is an effective method to display meta-analytic results. Given that how information is presented depends greatly on how it is presented, (Kahneman and Tversky, 1984), we conducted an experiment to ask what display factors affect meta-analysis interpretation, and whether these factors differ for audiences of different levels of statistical sophistication
Objectives:
1. What effects do different display formats have on interpretation of meta-analytic results?
2. What basic components of meta-analysis affect interpretation of meta-analytic results?
3. How do people of different statistical backgrounds differ in their interpretation of meta-analysis?
Methods: Questionnaires were mailed to 80 members of the Canadian Cochrane Network. In addition to describing their statistical expertise and familiarity with meta-analyses, they scored four separate meta-analyses on the extent to which the results showed a treatment effect, and their confidence in their decision (both on 7-point scales). Four factors of the display were manipulated, including the two between-subjects factors of Type of Display (Graphical or Non-Graphical) and Number of Studies comprising the meta-analysis (7 or 14), and the two repeated measures of Effect Size (Small/Large) and Homogeneity (Homogeneous/Heterogeneous results).
Results:
1. Presenting graphical displays alone without accompanying numeric information elicits more extreme responses than presenting numeric information alone, or both types together F(3,72)=3.95, p<01.
2. Effect size and Homogeneity interact; responses particularly favour the treatment when displays depict both large Effect Size and Homogeneous results F( 1,72)= 15.47, p<.001.
3. People with different statistical backgrounds interpret meta-analyses differently; those with more training are better able to interpret Homogeneity information F(l,72)=5.71, p=.02, are more confident in their responses when the results are clear, and less confident when the results are not clear F(l,72)=4.89, p=.03
Discussion: This study provides the first empirical evidence that different components of meta-analytic displays have substantial impact on the interpretation of results.
Objectives:
1. What effects do different display formats have on interpretation of meta-analytic results?
2. What basic components of meta-analysis affect interpretation of meta-analytic results?
3. How do people of different statistical backgrounds differ in their interpretation of meta-analysis?
Methods: Questionnaires were mailed to 80 members of the Canadian Cochrane Network. In addition to describing their statistical expertise and familiarity with meta-analyses, they scored four separate meta-analyses on the extent to which the results showed a treatment effect, and their confidence in their decision (both on 7-point scales). Four factors of the display were manipulated, including the two between-subjects factors of Type of Display (Graphical or Non-Graphical) and Number of Studies comprising the meta-analysis (7 or 14), and the two repeated measures of Effect Size (Small/Large) and Homogeneity (Homogeneous/Heterogeneous results).
Results:
1. Presenting graphical displays alone without accompanying numeric information elicits more extreme responses than presenting numeric information alone, or both types together F(3,72)=3.95, p<01.
2. Effect size and Homogeneity interact; responses particularly favour the treatment when displays depict both large Effect Size and Homogeneous results F( 1,72)= 15.47, p<.001.
3. People with different statistical backgrounds interpret meta-analyses differently; those with more training are better able to interpret Homogeneity information F(l,72)=5.71, p=.02, are more confident in their responses when the results are clear, and less confident when the results are not clear F(l,72)=4.89, p=.03
Discussion: This study provides the first empirical evidence that different components of meta-analytic displays have substantial impact on the interpretation of results.