Article type
Year
Abstract
Background: Interventions aimed at improving evidence-based implementation of clinical evidence are intrinsically complex. Complexity of these interventions can be explained by process evaluation details that may be modifying the performance of interventions. Methods for systematically reviewing and analysing process evaluations alongside reviews are poorly developed. Complementing evidence from robust designs provided in process evaluations and communication with authors can make systematic reviews more informative.
Objectives: This study explored the development and application of methods for reviewing process evaluations and contact with study authors to a Cochrane Effective Practice and Organisation of Care systematic review (printed educational materials), specifically to: (1) compare methods of seeking additional information (process evaluations and contact with authors), (2) determine the relative value of including other sources of information in a systematic review, and to (3) develop guidance for those undertaking reviews of complex interventions.
Methods: We collected information on quality appraisal items and key process evaluation components for twenty-two studies from three different sources: (1) effectiveness reports (RCT, CBA, ITS), (2) process evaluations and (3) direct contact with study authors. We developed optimal search strategies for identifying process evaluations, a framework for appraising quality of process evaluations and a conceptual framework for conducting process evaluations alongside systematic reviews. The relative value of information retrieved from these three sources was compared and supplemented evidence in the systematic review.
Results: 18/22 study authors participated in process evaluation interviews. Only 6/22 studies published process evaluations and secondary research. Direct contact with study authors improved information on quality assessment of trials, description of studies and interpretation of study results. Citation searches produced 1605 citations resulting in 30 relevant citations, the same yield as hand searching and contacting study authors.
Conclusions: Direct communication with authors is worthwhile for providing process evaluation details, clarifying quality assessments criteria and citing related publications that may be missing from primary publications. Additional efforts required for citation index searches were not beneficial. Process evaluations alongside systematic reviews may not be worthwhile until more progress is made in conducting and publishing process evaluations.
This project was funded by the Canadian Coordinating Office for Health Technology Assessment.
Objectives: This study explored the development and application of methods for reviewing process evaluations and contact with study authors to a Cochrane Effective Practice and Organisation of Care systematic review (printed educational materials), specifically to: (1) compare methods of seeking additional information (process evaluations and contact with authors), (2) determine the relative value of including other sources of information in a systematic review, and to (3) develop guidance for those undertaking reviews of complex interventions.
Methods: We collected information on quality appraisal items and key process evaluation components for twenty-two studies from three different sources: (1) effectiveness reports (RCT, CBA, ITS), (2) process evaluations and (3) direct contact with study authors. We developed optimal search strategies for identifying process evaluations, a framework for appraising quality of process evaluations and a conceptual framework for conducting process evaluations alongside systematic reviews. The relative value of information retrieved from these three sources was compared and supplemented evidence in the systematic review.
Results: 18/22 study authors participated in process evaluation interviews. Only 6/22 studies published process evaluations and secondary research. Direct contact with study authors improved information on quality assessment of trials, description of studies and interpretation of study results. Citation searches produced 1605 citations resulting in 30 relevant citations, the same yield as hand searching and contacting study authors.
Conclusions: Direct communication with authors is worthwhile for providing process evaluation details, clarifying quality assessments criteria and citing related publications that may be missing from primary publications. Additional efforts required for citation index searches were not beneficial. Process evaluations alongside systematic reviews may not be worthwhile until more progress is made in conducting and publishing process evaluations.
This project was funded by the Canadian Coordinating Office for Health Technology Assessment.