Article type
Year
Abstract
Background: implementation of evidence on multi-component interventions requires detailed information about critical intervention features and contextual moderators. Qualitative Comparative Analysis (QCA) is a promising new review method that uses Boolean set theory to identify complex causal ‘recipes’ of intervention features and contextual moderators. Currently no tool exists to critically appraise such reviews.
Objectives: we will introduce a new tool ‘CARU-QCA’ (Critical Appraisal of Reviews Using QCA) for appraising the application of QCA in systematic reviews and outline its features.
Methods: we used three sources of evidence to develop CARU-QCA: key papers on the logic and procedures of QCA; existing systematic review appraisal tools; and published QCA reviews.
Results: QCA blends a deep, holistic understanding of interventions and their underlying theory, with computer-based mathematical procedures for examining patterns in the data. As such the CARU-QCA tool appraises 1) the qualitative ‘approach’; 2) the ‘technical procedures’ and 3) ‘reporting and transparency’. To appraise the ‘approach’ the tool assesses how reviewers have developed deep case-knowledge of interventions, as well as how they have identified and specified intervention theories. The tool examines whether reviewers have used their case knowledge and theory appropriately to identify and specify 1) sets of studies to compare based on their outcomes, and 2) the intervention and contextual factors that can potentially explain those differences. Appraisal of ‘technical procedures’ involves examining the conduct and reporting of the six key analytical stages: 1) tabulating data; 2) constructing and checking ‘truth tables’ to examine configurations of intervention features in relation to outcomes; 3) resolving contradictory patterns in the data; 4) using Boolean logic to simplify the expression of configurations; 5) examining ‘logical remainders’ (theorising what would happen in configurations that are logically possible but not represented in the data); 6) returning to the deep case-knowledge and underlying theory to interpret the empirical results. Finally the tool appraises the ‘reporting and transparency’ of QCA including decisions around the coding of conditions and outcomes, the thresholds used (e.g. consistency thresholds), and the assumptions made in creating the final intermediate solution.
Conclusions: CARU-QCA provides a transparent and standardised approach for critiquing and evaluating reviews using QCA; it will be of value to those seeking to implement the findings of QCA reviews and to review authors seeking guidance on how to employ QCA.
Patient or healthcare consumer involvement: patient and public involvement (PPI) can be hugely valuable in scoping the focus of QCA analyses and in interpreting their findings. As such the tool includes questions to enable appraisal of whether PPI was appropriately carried out.
Objectives: we will introduce a new tool ‘CARU-QCA’ (Critical Appraisal of Reviews Using QCA) for appraising the application of QCA in systematic reviews and outline its features.
Methods: we used three sources of evidence to develop CARU-QCA: key papers on the logic and procedures of QCA; existing systematic review appraisal tools; and published QCA reviews.
Results: QCA blends a deep, holistic understanding of interventions and their underlying theory, with computer-based mathematical procedures for examining patterns in the data. As such the CARU-QCA tool appraises 1) the qualitative ‘approach’; 2) the ‘technical procedures’ and 3) ‘reporting and transparency’. To appraise the ‘approach’ the tool assesses how reviewers have developed deep case-knowledge of interventions, as well as how they have identified and specified intervention theories. The tool examines whether reviewers have used their case knowledge and theory appropriately to identify and specify 1) sets of studies to compare based on their outcomes, and 2) the intervention and contextual factors that can potentially explain those differences. Appraisal of ‘technical procedures’ involves examining the conduct and reporting of the six key analytical stages: 1) tabulating data; 2) constructing and checking ‘truth tables’ to examine configurations of intervention features in relation to outcomes; 3) resolving contradictory patterns in the data; 4) using Boolean logic to simplify the expression of configurations; 5) examining ‘logical remainders’ (theorising what would happen in configurations that are logically possible but not represented in the data); 6) returning to the deep case-knowledge and underlying theory to interpret the empirical results. Finally the tool appraises the ‘reporting and transparency’ of QCA including decisions around the coding of conditions and outcomes, the thresholds used (e.g. consistency thresholds), and the assumptions made in creating the final intermediate solution.
Conclusions: CARU-QCA provides a transparent and standardised approach for critiquing and evaluating reviews using QCA; it will be of value to those seeking to implement the findings of QCA reviews and to review authors seeking guidance on how to employ QCA.
Patient or healthcare consumer involvement: patient and public involvement (PPI) can be hugely valuable in scoping the focus of QCA analyses and in interpreting their findings. As such the tool includes questions to enable appraisal of whether PPI was appropriately carried out.