Article type
Year
Abstract
Background:
Systematic review (SR) authors commonly encounter situations in which meta-analysis is not possible for some or all studies (e.g., when studies report only the direction of the effect). In a survey of published SRs of public health and health systems interventions, 60% used a method other than meta-analysis, but the methods were rarely specified (5%) and were limited to those described in Cochrane guidance. Greater understanding is needed of the range of other synthesis methods considered by authors and factors that influence their decisions.
Methods:
We interviewed 12 experienced SR authors, editors and methodologists. We presented four scenarios in which it was not possible to combine results from all studies in a meta-analysis. Scenarios varied in the number and size of included studies; completeness of the reported results (i.e., fully reported [effect estimate and measure of precision] and partially reported [e.g., only a statement of statistical significance]); and other factors (e.g., risk of bias). Participants described the methods they considered to summarise, synthesise and present the results; whether they would combine all available studies in a single synthesis; and how they would draw an overall conclusion. Transcripts were coded to develop a framework of synthesis methods and factors that influenced decisions.
Results:
Factors that influenced participants’ decisions included their views on the purpose of synthesis, existing beliefs about study results and synthesis methods, trust in the available data and the decision-making needs of end users. Participants differed in whether they would or would not meta-analyse some of the available results. Commonly, participants considered explicit or implicit vote counting, alongside or instead of meta-analysis, when it was not possible to meta-analyse all studies. Their description and interpretation of the results varied.
Conclusions:
A diverse range of methodological approaches and influencing factors were identified. Although synthesis methods other than meta-analysis and their rationale are often not described in SRs, they could be described explicitly in protocols and SRs to enhance transparency.
Patient, public and/or consumer involvement:
Patients were not involved in this research. These findings are relevant in promoting transparency of methods, which will inform future guidance to strengthen SRs.
Systematic review (SR) authors commonly encounter situations in which meta-analysis is not possible for some or all studies (e.g., when studies report only the direction of the effect). In a survey of published SRs of public health and health systems interventions, 60% used a method other than meta-analysis, but the methods were rarely specified (5%) and were limited to those described in Cochrane guidance. Greater understanding is needed of the range of other synthesis methods considered by authors and factors that influence their decisions.
Methods:
We interviewed 12 experienced SR authors, editors and methodologists. We presented four scenarios in which it was not possible to combine results from all studies in a meta-analysis. Scenarios varied in the number and size of included studies; completeness of the reported results (i.e., fully reported [effect estimate and measure of precision] and partially reported [e.g., only a statement of statistical significance]); and other factors (e.g., risk of bias). Participants described the methods they considered to summarise, synthesise and present the results; whether they would combine all available studies in a single synthesis; and how they would draw an overall conclusion. Transcripts were coded to develop a framework of synthesis methods and factors that influenced decisions.
Results:
Factors that influenced participants’ decisions included their views on the purpose of synthesis, existing beliefs about study results and synthesis methods, trust in the available data and the decision-making needs of end users. Participants differed in whether they would or would not meta-analyse some of the available results. Commonly, participants considered explicit or implicit vote counting, alongside or instead of meta-analysis, when it was not possible to meta-analyse all studies. Their description and interpretation of the results varied.
Conclusions:
A diverse range of methodological approaches and influencing factors were identified. Although synthesis methods other than meta-analysis and their rationale are often not described in SRs, they could be described explicitly in protocols and SRs to enhance transparency.
Patient, public and/or consumer involvement:
Patients were not involved in this research. These findings are relevant in promoting transparency of methods, which will inform future guidance to strengthen SRs.