Article type
Year
Abstract
Background:
Discordant conclusions between meta-analyses occur even when meta-analysts are presented identical data, suggesting subjectivity is inherent in the “objective process”.
Objective:
To better understand the subjectivity in the meta-analytic interpretation process.
Methods:
We recruited 14 authors of meta-analyses and systematic reviews using theoretical sampling based on gender, training and experience. We asked participants about their general meta-analytical approach, and then provided simulated data sets (based on published meta-analyses) for treatments on four conditions. They were then asked to provide treatment effect estimates and confidence intervals after 3, 5, 10, and 20 studies for each condition. We used grounded theory methodology and semi-structured interviews to render explicit the analytical processes employed.
Results:
Our results suggest moral considerations shaped participants thinking at each stage of the interpretive process. (1) operationalization of the ‘should’, meta-analysts scrutinized the data pool. Many participants expressed strong moral reservations about certain data and questioned whether one ‘ought’ to conduct a meta-analysis; (2) examination of aggregate treatment effects, meta-analysts managed a ‘part-whole’ tension and shifted between focusing on individual studies and the entire dataset, with some displaying a moral commitment to maintain the whole dataset integrity; (3) translation of interpretation of data into action, participants deliberated about making a treatment decision, including moral concerns about financial resources invested, invoking “do no harm”, and a sympathy towards the patient population. These concerns sometimes led to a refusal to assign a point estimate, make a treatment decision, or withdraw from the meta-analytical process.
Conclusion:
This research highlights how data aggregation involves important moral considerations that are commonly concealed in ‘objectivist’ characterizations of meta-analysis. Understanding these processes should improve our understanding why different conclusions are drawn by different authors, and improve transparency for consumers of meta-analyses.
Discordant conclusions between meta-analyses occur even when meta-analysts are presented identical data, suggesting subjectivity is inherent in the “objective process”.
Objective:
To better understand the subjectivity in the meta-analytic interpretation process.
Methods:
We recruited 14 authors of meta-analyses and systematic reviews using theoretical sampling based on gender, training and experience. We asked participants about their general meta-analytical approach, and then provided simulated data sets (based on published meta-analyses) for treatments on four conditions. They were then asked to provide treatment effect estimates and confidence intervals after 3, 5, 10, and 20 studies for each condition. We used grounded theory methodology and semi-structured interviews to render explicit the analytical processes employed.
Results:
Our results suggest moral considerations shaped participants thinking at each stage of the interpretive process. (1) operationalization of the ‘should’, meta-analysts scrutinized the data pool. Many participants expressed strong moral reservations about certain data and questioned whether one ‘ought’ to conduct a meta-analysis; (2) examination of aggregate treatment effects, meta-analysts managed a ‘part-whole’ tension and shifted between focusing on individual studies and the entire dataset, with some displaying a moral commitment to maintain the whole dataset integrity; (3) translation of interpretation of data into action, participants deliberated about making a treatment decision, including moral concerns about financial resources invested, invoking “do no harm”, and a sympathy towards the patient population. These concerns sometimes led to a refusal to assign a point estimate, make a treatment decision, or withdraw from the meta-analytical process.
Conclusion:
This research highlights how data aggregation involves important moral considerations that are commonly concealed in ‘objectivist’ characterizations of meta-analysis. Understanding these processes should improve our understanding why different conclusions are drawn by different authors, and improve transparency for consumers of meta-analyses.