Background and methods: It has been considered that network meta-analyses (NMAs) would be the next generation evidence synthesis toolkit, which, when properly applied, could serve decision-making better than the conventional pairwise meta-analysis. However, NMAs are subject to similar methodological risks as standard pairwise systematic reviews. Because of their methodological complexity, it is probable that NMAs may be more vulnerable to such risks. Therefore, it is important to assess the quality of published NMAs before their results are implemented into clinical or public health practice. Currently, there is no consensus about how to assess the methodological quality of NMAs. AMSTAR (A MeaSurement Tool to Assess systematic Reviews) is widely used to evaluate the scientific quality of traditional systematic reviews or meta-analyses, but it is still unclear whether AMSTAR can be applied to NMAs. Therefore, we applied AMSTAR to NMAs in the field of cancer, and reported our experience in terms of applicability, reliability and feasibility.

Results: From 6408 citations retrieved, we identified and included 102 NMAs in the field of cancer. The inter-rater reliability was high, albeit items 1 (provide an ‘a priori’ design), 8 (scientific quality used in formulating conclusions), and 9 (appropriate method to combine studies) scored as 'moderate'. However, there was a high heterogeneity between the two pairs of reviewers. In terms of feasibility, a modified AMSTAR should be considered to apply to NMAs, especially regarding items 1 and 9; each review taking 10 to 20 minutes to complete.

Conclusions: Revisions and extensions of AMSTAR might be considerable to apply to NMAs.