Assessing quality in systematic reviews of the effective­ness of health promotion and public health: areas of consensus and dissension

Article type
Authors
Shepherd J
Abstract
Background: There have been very few published qualitative research studies in which systematic reviewers have been asked to give their views on review methodology, discuss the barriers and facilitators to effective reviewing, and describe their experiences of learning review skills. Objectives: To assess current approaches to, and general views on, methods for conducting systematic reviews of effectiveness in health promotion/public health (HP/PH), and to identify areas of consensus and dissension around the choice of techniques, methods and criteria employed. Methods: Semi-structured interviews with a purposive sample of 17 systematic reviewers, each of which had conducted at least one review of a HP/PH topic. Results: The prevailing view was that there was probably little or no consensus about quality assessment criteria in HP/PH. In assessing quality, subjective judgment was often required, contrary to the popular belief that systematic reviews are wholly ‘objective’. Barriers to effective systematic reviewing included: lack of time and resources; complexity of some HP/PH interventions; inclusion of observational evaluation designs; and poor reporting of primary studies. Some successful strategies to deal with these challenges were discussed. Interviewees learned systematic review skills through a combination of training, support from colleagues and mentors, literature and a strong emphasis on hands-on learning. Understanding statistics and critical appraisal were particularly challenging. Conclusions: Systematic reviewing in HP/PH is often challenging due the complexity of interventions and evaluation designs. This places additional demands on reviewers in terms of knowledge and skills required, often exacerbated by finite time scales and limited funding. Methodological pluralism is evident, although initiatives are in place, including efforts by the Cochrane Collaboration, to foster shared ways of working. However, the extent to which complete consensus is achievable in a multi-disciplinary field is questionable. A number of recommendations for research, policy and practice will be made.