A pilot Mixed Methods Appraisal Tool (MMAT) for systematic mixed studies reviews

Article type
Pluye P1, Cargo M2, Robert E3, Bartlett G1, O’Cathain A4, Griffiths F5, Boardman F5, Gagnon M6, Rousseau M3
1McGill University, Canada
2University of South Australia, Australia
3Université de Montréal, Canada
4University of Leeds, UK
5University of Warwick, UK
6Université Laval, Canada
Background: A new form of literature review has emerged, mixed studies review. These reviews include qualitative, quantitative and mixed methods studies. The Mixed Methods Appraisal Tool (MMAT) is intended to be used as a critical appraisal tool for systematic mixed studies reviews (SMSRs). In SMSRs, reviewers appraise studies that use different methods allowing them to obtain in-depth answers to complex research questions. However, appraising the quality of studies with different methods remains challenging. To facilitate this, the MMAT has been developed. The MMAT has been pilot tested using 32 evaluation studies for efficiency and reliability. On average, it took 14 minutes to appraise a study. Agreement between reviewers was moderate to perfect with regards to MMAT criteria, and substantial with respect to the overall quality score of appraised studies. Content validation: The current 2011 version of the MMAT is based on a literature review of MSRs, a mixed methods framework, and has been revised using feedback from four 90-minute workshops. The workshops involved diverse audiences such as graduate students enrolled in a mixed methods research course, researchers and research professionals with experience in qualitative, quantitative, and mixed methods research, and members of the Cochrane collaboration with experience in SMSRs or in systematic reviews of qualitative studies, e.g., meta-ethnography, or of quantitative studies, e.g., systematic reviews of randomized controlled trials.

Results: The current 2011 version of the MMAT (checklist and tutorial) contains two screening questions, and 19 questions corresponding to four types of studies (methodological domains): qualitative research, randomized controlled trials, non-randomized studies, quantitative descriptive, and mixed methods studies (http:// mixedmethodsappraisaltoolpublic.pbworks.com).

Conclusion: The MMAT is validated and unique for appraising the most common types of study designs, including mixed methods studies. The development of the MMAT is supported by an international research team with expertise on qualitative, quantitative and mixed methods research.