A Systematic Analysis of Process Evaluations

Tags: Oral
Weston R, Harden A, Oakley A

Introduction: There is an urgent need to address the lack of consensus on systematic criteria for assessing qualitative evaluations as part of systematic reviews and effectiveness research. Process evaluations, in particular, often illuminate the design, methodology and outcomes of experimental studies. When they do not, or there is no process element, then what we learn from such an assessment may help in improving the design and methodology of intervention research. We have recently completed a review of Appropriateness and Effectiveness of Peer-led Health Promotion Interventions for Young People. Part of this review consists of a systematic analysis of process evaluations (n=74). Twenty-five of these met the initial selection criteria for inclusion in a full systematic analysis, by two independent reviewers, using both the EPIC Review Guidelines and the EPIC database (Oakley, Peersman and Oliver, 1998). The criteria identified from different qualitative researchers/authors (Oakley, 1999 in press; Popay, 1997; Weston, 1996) will be used to reassess these surviving .

Objectives: To assess the appropriateness of criteria for systematically analysing qualitative/ process research/evaluation as part of evidence-based practice.

Methods: The criteria used to select a subset of studies (n=25) were developed by Weston (1996) for the European Union Cancer Prevention Review and have been used in the update of this review for the Health Education Authority (1999, in press). These are:

1. An explicit account of the theoretical underpinning/generic framework

2. A formative evaluation (needs assessment phase and pilot)

3. An intermediate evaluation: a review at the midpoint of the intervention

4. An impact (short term evaluation)

5. An outcome (long term evaluation)

6. And added value of existing Knowledge and practice

Oakley, (1999) has systematically compared 7 sets of criteria and formulated the following criteria (amalgamation of criteria across all authors). These are that qualitative research should have:

1. An explicit account of theoretical framework and inclusion of literature review if appropriate

2. Stated clear aims and objectives

3. A clear description of context

4. A clear description of sample

5. A clear description of fieldwork methods

6. An analysis of data by more than one researcher - systematic data collection

7. Sufficient original evidence to mediate between evidence and interpretation

Following this selection 15 studies were analysed by two reviewers. This process is currently underway and so the number of studies found to be sound (and agreed by both reviewers) is unknown at this point although the results will be available by mid-June. Oakley is not suggesting that these are necessarily adequate and this paper will describe this attempt to see how useful these might be. The results of this research will be disseminated via the Cochrane Collaboration and the Health Promotion and Public Health Field and journal articles. The conclusions will critically assess the contribution of this research to the current field of knowledge and methodological debate and suggest what the future research needs might be in terms of developing consensus on the criteria for systematically assessing such research.

Results: Available June 1999

Discussion: To be given at Rome Conference