Should we exclude poorly reported qualitative studies from systematic reviews? An evaluation of three reviews of qualitative data

Article type
Authors
Caroll C1, Booth A1, Lloyd-Jones M1
1ScHARR, University of Sheffield, Sheffield, UK
Abstract
Background: Ongoing debate persists regarding whether assessing the quality of qualitative studies included in systematic reviews is valid or meaningful. Objectives: This research aimed to determine whether the exclusion of qualitative studies based on key elements of reporting had any effect on the results of a synthesis. Methods: Three systematic reviews of qualitative data were performed on topics relating to public health and education. All included studies were appraised by two reviewers using criteria from Dixon-Woods’ key prompts for assessing qualitative studies (QSHC 2004), i.e. whether the sampling and methods of data collection and analysis were clearly described. Better reported studies provided clear details on all or most of these criteria. By contrast relatively less well-reported studies gave few or no such details. Thematic analysis was used as the method of synthesis in two reviews, and a version of framework synthesis in the third. In each case, a sensitivity analysis was performed in which relatively poorly reported studies were excluded, and an assessment was made about whether and how the synthesis was affected by the exclusion of these studies. Results: On the basis of quality of reporting described above, 1/20, 9/20 and 10/19 studies were excluded from the three reviews. By excluding these studies, only one review was affected, with only a single major theme and its two related sub-themes excluded from the synthesis. Otherwise, these exclusions did not affect the synthesis findings in a meaningful way. Conclusions: An evaluation of the findings of three systematic reviews indicates that the exclusion of poorly-described qualitative studies does not appear to affect either the overall results of such systematic reviews, or the richness of their findings. Excluding such studies from the data extraction and synthesis processes would save time and, potentially, also enhance the internal validity of the findings of systematic reviews of qualitative data.