Article type
Year
Abstract
Background: Program theory can provide reviewers with a structured approach for outlining the action model(s) (i.e., characteristics of implementer, implementing system, ecological context, partnering organisations, participants, intervention), change model(s) describing the intended sequence of proximal to distal outcomes in participants, and identifying relevant process evaluation variables (i.e., fidelity, dose). Integrating this information can strengthen attributions of effectiveness and inform knowledge translation.
Objectives: To assess the extent to which and how reviewers account for aspects of the action model, process evaluation measures and change model(s) underpinning their reviews.
Methods: Twenty-seven systematic effectiveness reviews on social welfare, criminal justice and education published by the Campbell Collaboration (some co-registered with Cochrane) with reported outcomes on children or youth were reviewed. An assessment tool, informed by a program theory approach, was developed to assess elements of the action model, change model and process evaluation measures within reviews. Two raters completed extractions using a 7-point rating scale and noting comments in EPPI-Reviewer Version 4.0.
Results: Within the action model, the majority of reviews accounted for one or more aspects of the intervention (i.e., format) and participant characteristics with aspects of the implementing organisation, implementer and partnerships considered less often. Overall, action models could have been more clearly defined. Intervention dose was the most common process evaluation measure. For the change model, most reviews were 'black box’ with change processes remaining undifferentiated. Reviewers expressed interest in identifying active ingredients of interventions and integrating qualitative evidence. This presentation will feature exemplars to illustrate how program theory can be integrated into reviews to understand heterogeneity.
Conclusions: Program theory can add value to reviews by informing the review objectives, inclusion/exclusion criteria, search strategies, and the operationalisation/analysis of variables related to the action model(s), change model(s) and process evaluation. Resources, training, and guidance are needed to support reviewers.
Objectives: To assess the extent to which and how reviewers account for aspects of the action model, process evaluation measures and change model(s) underpinning their reviews.
Methods: Twenty-seven systematic effectiveness reviews on social welfare, criminal justice and education published by the Campbell Collaboration (some co-registered with Cochrane) with reported outcomes on children or youth were reviewed. An assessment tool, informed by a program theory approach, was developed to assess elements of the action model, change model and process evaluation measures within reviews. Two raters completed extractions using a 7-point rating scale and noting comments in EPPI-Reviewer Version 4.0.
Results: Within the action model, the majority of reviews accounted for one or more aspects of the intervention (i.e., format) and participant characteristics with aspects of the implementing organisation, implementer and partnerships considered less often. Overall, action models could have been more clearly defined. Intervention dose was the most common process evaluation measure. For the change model, most reviews were 'black box’ with change processes remaining undifferentiated. Reviewers expressed interest in identifying active ingredients of interventions and integrating qualitative evidence. This presentation will feature exemplars to illustrate how program theory can be integrated into reviews to understand heterogeneity.
Conclusions: Program theory can add value to reviews by informing the review objectives, inclusion/exclusion criteria, search strategies, and the operationalisation/analysis of variables related to the action model(s), change model(s) and process evaluation. Resources, training, and guidance are needed to support reviewers.