Article type
Year
Abstract
Background: The quality of studies published as systematic reviews/meta-analyses (SR/MAs) in many fields of health care is low. AMSTAR-2 (‘A Measurement Tool to Assess Systematic Reviews’) and ROBIS (‘Risk of Bias in Systematic Reviews’) are independent instruments that focus on different but overlapping aspects of SR/MA quality.
Objectives: We aimed to assess and compare the quality of studies published as SR/MAs on nutritional interventions in cancer prevention using AMSTAR-2 and ROBIS.
Methods: We performed a systematic survey of MEDLINE, EMBASE and the Cochrane Library for SR/MAs published between January 2010 and August 2018. We included studies identified as SR/MAs in the title and/or abstract which examined the effects of any nutritional intervention for cancer prevention. Following a calibration exercise, study selection, data extraction and AMSTAR-2/ROBIS assessments were completed by two independent reviewers with conflicts resolved by discussion or consultation with a third reviewer. AMSTAR-2 is comprised of 16 questions for which yes, partial yes, no, or not applicable can be applied. ROBIS consists of 21 signaling questions for which yes, partial yes, partial no, no, or no information can be applied. Our protocol was registered in PROSPERO: CRD42019121116.
Results: Out of 743 included articles, we selected a random sample of 101 for detailed analyses. Overall, the quality of SR/MAs in nutrition was low on AMSTAR-2 and ROBIS. We made 11 comparisons between AMSTAR-2 and ROBIS questions assessing the same construct. Some questions measuring separate constructs could not be compared (i.e. explanation for selection of study design in AMSTAR-2; appropriateness of eligibility criteria in ROBIS), and in some cases multiple questions were combined for comparison (i.e. comprehensiveness of searches and validity of statistical methods used). Comparability of reviewers’ judgments between AMSTAR-2 and ROBIS questions is presented in Table 1. In 9 out of 11 comparisons the assessments were comparable, ranging from 78.2%-99.0% agreement. For 2 comparisons: including comprehensive literature search and publication bias, the assessments were poorly comparable (59.4%).
Conclusions: AMSTAR-2 and ROBIS instruments mostly address similar aspects of SR/MA quality and our assessments using different instruments were similar. However, AMSTAR-2 uniquely addresses reporting of excluded studies, sources of funding, conflict of interest within individual studies and reasons for selection of study designs for inclusion, while ROBIS uniquely addresses adherence to predefined analyses, appropriateness and restrictions within eligibility criteria. Potential users should be aware of the considerably large overlap and the small but unique differences.
Patient or healthcare consumer involvement: We did not have a patient or healthcare consumer involved, however information about the quality of SR/MAs is important for consumers when interpreting results.
Funding: Project funded by National Science Centre, No. UMO-2017/25/B/NZ7/01276
Objectives: We aimed to assess and compare the quality of studies published as SR/MAs on nutritional interventions in cancer prevention using AMSTAR-2 and ROBIS.
Methods: We performed a systematic survey of MEDLINE, EMBASE and the Cochrane Library for SR/MAs published between January 2010 and August 2018. We included studies identified as SR/MAs in the title and/or abstract which examined the effects of any nutritional intervention for cancer prevention. Following a calibration exercise, study selection, data extraction and AMSTAR-2/ROBIS assessments were completed by two independent reviewers with conflicts resolved by discussion or consultation with a third reviewer. AMSTAR-2 is comprised of 16 questions for which yes, partial yes, no, or not applicable can be applied. ROBIS consists of 21 signaling questions for which yes, partial yes, partial no, no, or no information can be applied. Our protocol was registered in PROSPERO: CRD42019121116.
Results: Out of 743 included articles, we selected a random sample of 101 for detailed analyses. Overall, the quality of SR/MAs in nutrition was low on AMSTAR-2 and ROBIS. We made 11 comparisons between AMSTAR-2 and ROBIS questions assessing the same construct. Some questions measuring separate constructs could not be compared (i.e. explanation for selection of study design in AMSTAR-2; appropriateness of eligibility criteria in ROBIS), and in some cases multiple questions were combined for comparison (i.e. comprehensiveness of searches and validity of statistical methods used). Comparability of reviewers’ judgments between AMSTAR-2 and ROBIS questions is presented in Table 1. In 9 out of 11 comparisons the assessments were comparable, ranging from 78.2%-99.0% agreement. For 2 comparisons: including comprehensive literature search and publication bias, the assessments were poorly comparable (59.4%).
Conclusions: AMSTAR-2 and ROBIS instruments mostly address similar aspects of SR/MA quality and our assessments using different instruments were similar. However, AMSTAR-2 uniquely addresses reporting of excluded studies, sources of funding, conflict of interest within individual studies and reasons for selection of study designs for inclusion, while ROBIS uniquely addresses adherence to predefined analyses, appropriateness and restrictions within eligibility criteria. Potential users should be aware of the considerably large overlap and the small but unique differences.
Patient or healthcare consumer involvement: We did not have a patient or healthcare consumer involved, however information about the quality of SR/MAs is important for consumers when interpreting results.
Funding: Project funded by National Science Centre, No. UMO-2017/25/B/NZ7/01276