Including process evaluations in systematic reviews: reflections from a review of sexual health promotion

Article type
Authors
Harden A, Barnett-Page E, Shepherd J, Kavanagh J, Picot J
Abstract
Background: There is much interest in including process evaluations in systematic reviews, especially reviews of complex interventions. The methodology for including process data within reviews is still emerging and there are few worked examples. Objectives: To reflect on the methods used and the added value of including process evaluations in a systematic review of the effectiveness of school-based skill-building interventions amongst young people to prevent sexually transmitted infections. Methods: The review included 12 good quality RCTs of which nine included a process evaluation. Standard systematic review methods, including meta-analysis, were use to synthesise effect sizes from trials. For the process evaluations, two reviewers independently extracted data and appraised quality according to seven criteria that covered rigour in methods and depth and breadth of findings. A four stage narrative synthesis was conducted: i) preparation of evidence tables; ii) initial theme generation; iii) further development and translation of themes across studies; and iv) finalising the synthesis. Results: Process evaluations varied in quality. Emphasis was placed on studies judged to be of at least medium quality. The final synthesis revealed two sets of factors that explained the limited impact of interventions found in the effectiveness synthesis: difficulties in intervention implementation and lack of intervention acceptability. The production and rigour of the final synthesis were greatly facilitated by a discussion of the initial set of themes by all review authors in relation to the findings of the effectiveness synthesis. Conclusions: This worked example suggests that systematic inclusion of process data can add considerable value to reviews by providing robust, empirically based explanations of observed effect sizes and hypotheses to test in new research. Questions remain, however, around the methodological quality, depth and reporting of process evaluations, the additional resources required, and the relative value of different methods of quality assessing and synthesising findings.