Article type
Year
Abstract
Background: When meta-analysts arrive at discrepant conclusions from different meta-analyses, methodological differences are assumed to be the primary reason for the variation. Our research team has previously shown that discordant interpretations can exist even when meta-analysts are presented with identical data.
objective: To better understand the analytic processes meta-analysts employ when interpreting meta-analyses.
Methods: For this mixed-methods study, we recruited authors of meta-analyses/systematic reviews using theoretical sampling and the following variables: gender, professional training, number of meta-analyses conducted. Fourteen participants have been recruited to date. After asking participants about their general meta-analytical approach, they were provided with simulated data sets (based on published meta-analyses) in relation to treatments for four different conditions. They were then asked to provide treatment effect estimates and confidence intervals after 3, 5, 10, and 20 studies for each of the four conditions. Using grounded theory methodology, we conducted semi-structured qualitative interviews to render explicit the different analytical processes employed.
Results: Preliminary findings suggest that meta-analysts engage in the following analytical processes: 1) scrutinizing the data for 'poolability’ 2) managing a 'part-whole’ tension (i.e., shifting between a focus on individual studies and the entire dataset); and 3) determining a treatment decision. Participants demonstrated variation within these processes as well as in the micro-processes they used to frame and formulate interpretations. Finally, we describe moral dimensions that shape meta-analysts’ analytic processes including: a desire to maintain the integrity of the whole; refusing to recommend a treatment or assign a point estimate because it would not be 'right’ a concern about financial resources required to further evaluate a treatment; and complying with external standards.
Conclusion: These results demonstrate that meta-analysts engage in subjective analytical processes, including important moral considerations, when interpreting meta-analyses. These processes should be considered to improve the format of meta-analysis presentation.
objective: To better understand the analytic processes meta-analysts employ when interpreting meta-analyses.
Methods: For this mixed-methods study, we recruited authors of meta-analyses/systematic reviews using theoretical sampling and the following variables: gender, professional training, number of meta-analyses conducted. Fourteen participants have been recruited to date. After asking participants about their general meta-analytical approach, they were provided with simulated data sets (based on published meta-analyses) in relation to treatments for four different conditions. They were then asked to provide treatment effect estimates and confidence intervals after 3, 5, 10, and 20 studies for each of the four conditions. Using grounded theory methodology, we conducted semi-structured qualitative interviews to render explicit the different analytical processes employed.
Results: Preliminary findings suggest that meta-analysts engage in the following analytical processes: 1) scrutinizing the data for 'poolability’ 2) managing a 'part-whole’ tension (i.e., shifting between a focus on individual studies and the entire dataset); and 3) determining a treatment decision. Participants demonstrated variation within these processes as well as in the micro-processes they used to frame and formulate interpretations. Finally, we describe moral dimensions that shape meta-analysts’ analytic processes including: a desire to maintain the integrity of the whole; refusing to recommend a treatment or assign a point estimate because it would not be 'right’ a concern about financial resources required to further evaluate a treatment; and complying with external standards.
Conclusion: These results demonstrate that meta-analysts engage in subjective analytical processes, including important moral considerations, when interpreting meta-analyses. These processes should be considered to improve the format of meta-analysis presentation.