Background: One of the underlying principles of meta-analysis is that sufficiently similar trials are aggregated in order to gain greater precision of effect size estimate and confidence around that estimate. This assumption breaks down outside clinical settings where interventions tend to be complicated and complex; they are also rarely replicated. In order to identify what works for whom in what situation, we need methods that ‘aggregate’ findings statistically and ‘configure’ (i.e. arrange, or compare and contrast) findings from different contexts.
Objectives: To investigate the utility of a range of methods that might be useful for complex topics that lie on different points of the ‘aggregate—configure continuum’.
Methods: We consider four methods that use quantitative or semi-quantitative approaches to synthesising quantitative data for different types of research questions: a) Multivariate meta-analysis; b) Path analysis, including structural equation modelling; c) Qualitative comparative analysis; and d) Factor analysis, including principal components analysis. We discuss the conceptual basis and potential limitations of each. Examples are provided from a recent systematic review on a complex public health topic.
Results: In addition to aggregation to determine a point estimate, we identified the following types of analytical questions that might be considered by systematic reviewers as relates to the synthesis of quantitative data: a) Identifying confounding variables; b) The personality of studies (i.e., profiles of interventions); c) Defining theoretical constructs using indicator variables; and d) Construct validation. The four methods discussed show promise in addressing these different research questions, which lie on different points of the aggregate-configure continuum. In our example systematic review, we were able to address a broader range of research questions.
Conclusions: Meta-analysis alone is insufficient in some situations when variation is encountered—and is indeed part of the purpose of the analysis. While there are alternatives, they each have particular limitations, and further methodological development is required.